Virtually Anything

Using kvm images with bhyve

2018-04-17T22:00:00+00:00

So far, most of my use of bhyve has been with kvm images that have been customize to work better with bhyve. For Ubuntu images, those customizations have generally been:

Configure grub to use the serial console
Install a version of cloud-init that has some fixes

So long as nothing is putting garbage on the second serial line, the version of cloud-init that comes with Ubuntu works well enough that it can be used to configure the serial console. Early on, I was having troubles with garbage on the serial line, but that seems to have been an ephemeral issue.

Cloud-init supports user-data with a variety of content. I’ll use cloud-config to accomplish what I’m after. This is not as straight-forward as we would like - ideally we would just splat some more json in customer_metadata. But cloud-init really likes yaml. And even if it didn’t, customer_metadata does not support nested structures. So, we need some yaml wrapped in json.

The cloud-config that’s needed looks like this:

#cloud-config
write_files:
  - content: |
      GRUB_DEFAULT=0
      GRUB_TIMEOUT=5
      GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
      GRUB_CMDLINE_LINUX_DEFAULT="tsc=reliable earlyprintk"
      GRUB_CMDLINE_LINUX=""
      GRUB_TERMINAL="serial console"
      GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200n8"
      GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"
    path: /etc/default/grub
runcmd:
  - update-grub2 >/var/tmp/update-grub2.log 2>&1
  - shutdown -r +1

To get that into json, I use a modified version of a helper script found in the cloud-init docs.

#!/usr/bin/python

import json
import sys

from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText

if len(sys.argv) == 1:
    print("%s input-file:type ..." % (sys.argv[0]))
    sys.exit(1)

combined_message = MIMEMultipart()
for i in sys.argv[1:]:
    (filename, format_type) = i.split(":", 1)
    with open(filename) as fh:
        contents = fh.read()
    sub_message = MIMEText(contents, format_type, sys.getdefaultencoding())
    sub_message.add_header('Content-Disposition', 'attachment; filename="%s"' % (filename))
    combined_message.attach(sub_message)

print(json.dumps({'cloud-init:user-data': str(combined_message)}))

The customer_metadata fodder is generated with:

$ userdatagen /tmp/kvm-to-bhyve.yaml:cloud-config
{"cloud-init:user-data": "From nobody Tue Apr 17 16:40:55 2018\nContent-Type: multipart/mixed; boundary=\"===============8760689715383894136==\"\nMIME-Version: 1.0\n\n--===============8760689715383894136==\nMIME-Version: 1.0\nContent-Type: text/cloud-config; charset=\"us-ascii\"\nContent-Transfer-Encoding: 7bit\nContent-Disposition: attachment; filename=\"/tmp/kvm-to-bhyve.yaml\"\n\n#cloud-config\nwrite_files:\n  - content: |\n      GRUB_DEFAULT=0\n      GRUB_TIMEOUT=5\n      GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`\n      GRUB_CMDLINE_LINUX_DEFAULT=\"tsc=reliable earlyprintk\"\n      GRUB_CMDLINE_LINUX=\"\"\n      GRUB_TERMINAL=\"serial console\"\n      GRUB_CMDLINE_LINUX=\"console=tty0 console=ttyS0,115200n8\"\n      GRUB_SERIAL_COMMAND=\"serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1\"\n    path: /etc/default/grub\nruncmd:\n  - update-grub2 >/var/tmp/update-grub2.log 2>&1\n  - shutdown -r +1\n\n--===============8760689715383894136==--\n"}

That gets copied into the customer_metadata section of my vmadm payload.

{
  "alias": "kvm2bhyve",
  "brand": "bhyve",
  "resolvers": [
    "8.8.8.8",
    "8.8.4.4"
  ],
  "ram": "1024",
  "vcpus": "2",
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "10.88.88.207",
      "netmask": "255.255.255.0",
      "gateway": "10.88.88.2",
      "model": "virtio",
      "primary": true
    }
  ],
  "disks": [
    {
      "image_uuid": "429bf9f2-bb55-4c6f-97eb-046fa905dd03",
      "boot": true,
      "model": "virtio"
    }
  ],
  "zlog_max_size": 1048576,
  "zlog_keep_rotated": 5,
  "customer_metadata": {
    "cloud-init:user-data": "From nobody Tue Apr 17 16:40:55 2018\nContent-Type: multipart/mixed; boundary=\"===============8760689715383894136==\"\nMIME-Version: 1.0\n\n--===============8760689715383894136==\nMIME-Version: 1.0\nContent-Type: text/cloud-config; charset=\"us-ascii\"\nContent-Transfer-Encoding: 7bit\nContent-Disposition: attachment; filename=\"/tmp/kvm-to-bhyve.yaml\"\n\n#cloud-config\nwrite_files:\n  - content: |\n      GRUB_DEFAULT=0\n      GRUB_TIMEOUT=5\n      GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`\n      GRUB_CMDLINE_LINUX_DEFAULT=\"tsc=reliable earlyprintk\"\n      GRUB_CMDLINE_LINUX=\"\"\n      GRUB_TERMINAL=\"serial console\"\n      GRUB_CMDLINE_LINUX=\"console=tty0 console=ttyS0,115200n8\"\n      GRUB_SERIAL_COMMAND=\"serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1\"\n    path: /etc/default/grub\nruncmd:\n  - update-grub2 >/var/tmp/update-grub2.log 2>&1\n  - shutdown -r +1\n\n--===============8760689715383894136==--\n"
  }
}

Notice that the image_uuid is for an ubuntu-certified-16.04 image.

If all goes well, a couple minutes after running vmadm create, you will notice the grub menu on the console.

Mutual exclusion of bhyve and kvm on SmartOS

2018-03-20T14:00:00+00:00

Note: This post is of historical interest only. With the fix for OS-7080 in July of 2018 bhyve and kvm can coexist in peace.

If two hypervisors are not able to coordinate with each other, they must not both use hardware viritualization. Within SmartOS, there is a pair of functions, hvm_excl_hold() and hvm_excl_rele(), described by this comment:

/*
 * HVM Exclusion Interface
 *
 * To avoid VMX/SVM conflicts from arising when multiple hypervisor providers
 * (eg. KVM, bhyve) are shipped with the system, this simple advisory locking
 * system is presented for their use.  Until a proper hypervisor API, like the
 * one in OSX, is shipped in illumos, this will serve as opt-in regulation to
 * dictate that only a single hypervisor be allowed to configure the system and
 * run at any given time.
 */

This advisory lock is taken as the first VM is starting and is released when the last one is stopped. This allows the use of either hypervisor, but prevents the concurrent use of both. Notably, the lock is not taken at module load time.

Who has the lock?

You can use mdb to see which hypervisor currently has the lock:

# mdb -k
Loading modules: [ ... ]
> hvm_excl_holder::print
0

After starting a kvm instance, it says:

> hvm_excl_holder::print
0xfffffffff8224185 "SmartOS KVM"

How is the lock taken and released?

Using a little dtrace, we can see the call stack involved in taking the lock. For some reason, dtrace couldn’t resolve the symbols so I used ::whatis in mdb to fill in the missing symbols.

# dtrace -n 'hvm_excl_*:entry { stack() }'
dtrace: description 'hvm_excl_*:entry ' matched 2 probes
CPU     ID                    FUNCTION:NAME
 48   8666              hvm_excl_hold:entry 
              0xfffffffff82aec97 (vmmdev_mod_incr+0x37)
              0xfffffffff82aee23 (vmmdev_do_vm_create+0x73)
              0xfffffffff82afbd3 (vmm_ioctl+0x193)
              genunix`cdev_ioctl+0x39
              specfs`spec_ioctl+0x60
              genunix`fop_ioctl+0x55
              genunix`ioctl+0x9b
              unix`sys_syscall+0x19f

 72   9030              hvm_excl_rele:entry 
              0xfffffffff82aed11 (vmmdev_mod_decr+0x31)
              0xfffffffff82af6b6 (vmm_do_vm_destroy_locked+0xe6)
              0xfffffffff82af70d (vmm_do_vm_destroy+0x3d)
              0xfffffffff82c9f9a (vmm_zsd_destroy+0x3a)
              genunix`zsd_apply_destroy+0x1d1
              genunix`zsd_apply_all_keys+0x5f
              genunix`zone_zsd_callbacks+0xe6
              genunix`zone_destroy+0x124
              genunix`zone+0x1e7
              unix`sys_syscall+0x19f

This release output shows that the bhyve instance I was running is in a zone and the destroy of the VM state was taken care of by the zone specific data destroy callback.

And now for something unexpected

Despite what I said above, it is possible to run bhyve and kvm instances simultaneously.

[root@emy-17 /root]# vmadm start $(vm test)
Successfully started VM 79062669-e229-e55d-960d-9b18d0fed8d0
[root@emy-17 /root]# vmadm start $(vm kvm-debian8)
Successfully started VM 57409636-927a-cc7a-d744-e0f92b73f4e4
[root@emy-17 /root]# vmadm list state=running
UUID                                  TYPE  RAM      STATE             ALIAS
57409636-927a-cc7a-d744-e0f92b73f4e4  KVM   1024     running           kvm-debian8
79062669-e229-e55d-960d-9b18d0fed8d0  BHYV  2048     running           test

It turns out that bhyve has the lock.

# echo hvm_excl_holder::print | mdb -k
0xfffffffff82caded "bhyve"

So, what’s happening here? In this case, kvm was not able to get the lock to use hardware virtualization. It is falling back to QEMU’s software virtualization, which is much slower.

Had the kvm instance been started first, kvm would have gotten the lock and the bhyve instance would have failed to boot. Now that both of these VMs are set to being enabled, on the next reboot of the host, they will race to get the lock.

To state the obvious, trying to run bhyve and kvm VMs simultaneously is not recommended.

OmniOS bhyve guest

2018-03-14T23:00:00+00:00

I had a very pleasant surprise today. I have some work that I need to contribute to illumos and figured that a little dogfooding would be good for me. It seems as though two excellent distros for building illumos are OpenIndiana and OmniOS. Since I don’t really need a desktop and I’ve not used OmniOS before, OmniOS seemed like a great choice for the day.

After a bit of looking around at omniosce.org I came across something that was almost too good to be true: omnios-bhyve.zfs.xz!

Let’s take that for a spin:

# curl https://downloads.omniosce.org/media/bloody/omnios-bhyve.zfs.xz | xzcat | zfs recv zones/omnios
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  343M  100  343M    0     0  3374k      0  0:01:44  0:01:44 --:--:-- 4368k
# zfs clone zones/omnios@good zones/o1
# bhyve -m 16g -c 16 -l com1,stdio -P -H -s 1,lpc -s 3,virtio-blk,/dev/zvol/rdsk/zones/o1 -s 4,virtio-net-viona,o1 -l bootrom,/usr/share/bhyve/uefi-csm-rom.bin o1

OMG! A few seconds later and I saw loader, and a few seconds after that I had a login prompt. Woot!

Well, let’s make this work with the SmartOS bits that are almost ready for you to take for a spin. First, I’ll turn this into an image that will work well with vmadm.

# zfs rename zones/omnios@good zones/omnios@final
# zfs send zones/omnios@final | pigz > /zones/images/6c4da93e-cffd-e809-a352-e4e11de3b067/disk0.zfs.gz
# ls -l disk0.zfs.gz
total 1011562
-rw-r--r--   1 root     root     517860460 Mar 15 00:20 disk0.zfs.gz
-rw-r--r--   1 root     root         918 Mar 15 00:19 manifest.json
# digest -a sha1 disk0.zfs.gz
9d14866b9ec35443adcffc34fed54bd7f1e62700
# vi manifest.json
# imgadm install -m manifest.json -f disk0.zfs.gz
Installing image 6c4da93e-cffd-e809-a352-e4e11de3b067 (omnios-master-39b8863d01@20180301)
...-e809-a352-e4e11de3b067 [=========================================>] 100% 493.87MB  29.32MB/s    16s
Installed image 6c4da93e-cffd-e809-a352-e4e11de3b067 (omnios-master-39b8863d01@20180301)
[root@emy-17 ~]# imgadm list os=illumos
UUID                                  NAME                      VERSION   OS       TYPE  PUB
6c4da93e-cffd-e809-a352-e4e11de3b067  omnios-master-39b8863d01  20180301  illumos  zvol  2018-03-15

Next, shamelessly copy an early vmadm create payload to create a new vm.

# cp test.json omni1.json
# vi omni1.json
# vmadm create -f omni1.json
[some output that I lost in my haste.  Just the normal stuff that says things went well.]
# vmadm console $(vm omni1)
...
Copyright (c) 1983, 2010, Oracle and/or its affiliates. All rights reserved.
Copyright (c) 2017-2018 OmniOS Community Edition (OmniOSce) Association.
Loading smf(5) service descriptions: 131/131
Configuring devices.
NOTICE: vioif0: Got MAC address from host: 72:93:42:ef:32:a0
NOTICE: Csum enabled.
Applying initial boot settings...
Hostname: omniosce

omniosce console login:

root@omniosce:~# dladm show-link
LINK        CLASS     MTU    STATE    BRIDGE     OVER
vioif0      phys      1500   up       --         --
root@omniosce:~# ipadm create-if vioif0
root@omniosce:~# ipadm create-addr -T static -a 172.26.17.173/24 vioif0/v4
root@omniosce:~# ping 172.26.17.1
172.26.17.1 is alive
root@omniosce:~# route -p add default 172.26.17.1
add net default: gateway 172.26.17.1: entry exists
add persistent net default: gateway 172.26.17.1
root@omniosce:~# vi /etc/resolv.conf
root@omniosce:~# vi /etc/nsswitch.conf
...

Thanks much to (andyf?) for creating the image.

Why must we manually configure a serial console?

2018-03-11T18:00:00+00:00

I was looking at ACPI tables in a bhyve guest and stumbled across this:

root@test:/tmp# acpidump -b spcr
root@test:/tmp# iasl -d spcr.dat

Intel ACPI Component Architecture
ASL+ Optimizing Compiler version 20160831-64
Copyright (c) 2000 - 2016 Intel Corporation

Input file spcr.dat, Length 0x50 (80) bytes
ACPI: SPCR 0x0000000000000000 000050 (v01 BHYVE  BVSPCR   00000001 BHYV 00000001)
Acpi Data Table [SPCR] decoded
Formatted output:  spcr.dsl - 2599 bytes
root@test:/tmp# cat spcr.dsl
/*
 * Intel ACPI Component Architecture
 * AML/ASL+ Disassembler version 20160831-64
 * Copyright (c) 2000 - 2016 Intel Corporation
 *
 * Disassembly of spcr.dat, Sun Mar 11 17:44:37 2018
 *
 * ACPI Data Table [SPCR]
 *
 * Format: [HexOffset DecimalOffset ByteLength]  FieldName : FieldValue
 */

[000h 0000   4]                    Signature : "SPCR"    [Serial Port Console Redirection table]
[004h 0004   4]                 Table Length : 00000050
[008h 0008   1]                     Revision : 01
[009h 0009   1]                     Checksum : 78
[00Ah 0010   6]                       Oem ID : "BHYVE "
[010h 0016   8]                 Oem Table ID : "BVSPCR  "
[018h 0024   4]                 Oem Revision : 00000001
[01Ch 0028   4]              Asl Compiler ID : "BHYV"
[020h 0032   4]        Asl Compiler Revision : 00000001

[024h 0036   1]               Interface Type : 00
[025h 0037   3]                     Reserved : 000000

[028h 0040  12]         Serial Port Register : [Generic Address Structure]
[028h 0040   1]                     Space ID : 01 [SystemIO]
[029h 0041   1]                    Bit Width : 08
[02Ah 0042   1]                   Bit Offset : 00
[02Bh 0043   1]         Encoded Access Width : 00 [Undefined/Legacy]
[02Ch 0044   8]                      Address : 00000000000003F8

[034h 0052   1]               Interrupt Type : 01
[035h 0053   1]          PCAT-compatible IRQ : 04
[036h 0054   4]                    Interrupt : 00000000
[03Ah 0058   1]                    Baud Rate : 07
[03Bh 0059   1]                       Parity : 00
[03Ch 0060   1]                    Stop Bits : 01
[03Dh 0061   1]                 Flow Control : 03
[03Eh 0062   1]                Terminal Type : 02

Notice that it seems to have the address for COM1 (0x3F8), IRQ 4, etc., which all seems legit for my setup. If all of this available when a machine redirects a serial console, why have I been manually configuring serial consoles all these years?

Let me predict the most common reasons:

It is somethign “new” that Microsoft came up with. By “new” I mean sometime in the past decade.
The BIOS implementations that support it are so buggy that the information is not reliable

FWIW, the specification is available in .docx format from Microsoft.

Socializing my blog

2018-03-04T16:00:00+00:00

There have been a few changes in this blog:

Now has an atom feed. Thanks to John Levon for the nudge and the Jekyll Feed Plugin for the simplicity.
Comments as github issues. Thanks to Ivan Zuzak and Michael Koohafkan.
Prefer to tweet? We got that too! Check out the recent mentions on the right-hand side and add your own. Sadly, the View on Twitter button seems to be useless.

uefi debug in bhyve

2018-02-26T02:00:00+00:00

While debugging a bug in SmartOS bhyve seen on Ubuntu guests, I needed to debug two areas that most people avoid: grub and uefi. Debugging grub is not so bad, as you can at least add print statements and they appear on the console. uefi was not quite as cooperative at first.

There are several things involved at build time:

While running the uefi-edk2 build script, you need the command line argument -DDEBUG_ON_SERIAL_PORT=TRUE. If building uefi-edk2 in illumos-extra, this is already taken care of.
You may need to tweak the PcdDebugPrintErrorLevel or PcdDebugPropertyMask in BhyvePkg/BhyvePkgX64.dsc to filter the particular types of messages you see. I’ve not experimented with these.
In BhyvePkg/Csm/BhyveCsm16/GNUmakefile, DEBUG_PORT needs to point to the proper COM port.
In BhyvePkg/Csm/BhyveCsm16/Printf.c, DebugLevel needs to be set to a level that gets the desired verbosity.

These last two bullets are taken care of with this changeset. Notice that it also changes PcdDebugIoPort in BhyvePkg/BhyvePkg.dec. I had expected that that would be sufficient to change the debug port to COM1, but it seemed to have no effect.

Once I did this, I wasn’t completely out of the woods, as I found that this caused bhyve to crash because its uart emulation only handled 1 or 2 byte operations. uefi works 4 bytes at a time. Here’s the fix.

By the time you are reading this, the bhyve bug will probably be long fixed. To save you the hassle of building a debug uefi image, you can snag the one I built. You can activate that for a single bhyve zone with:

cp .../uefi-csm-rom-debug-com1-info.bin /zones/$uuid/root
vmadm update $uuid bootrom=/uefi-csm-rom-debug-com1-info.bin

dtrace or mdb new processes

2018-02-19T01:57:23+00:00

I have a process that will be started later in a special context that I need to trace with truss(1M). An easy way to do this is with a little help from dtrace:

In real life I did this as a one-liner on the command line. I expand it here for the sake of comments

#! /usr/sbin/dtrace -ws

/*
 * Watch for the first system call performed by the zhyve program and
 * then trace that program with truss.
 */
syscall:::entry
/execname == "zhyve"/
{
	/* Stop the program */
	stop();
        /* Use truss to start tracing the program.  This causes the program to continue. */
	system("truss -t all -v all -w all -p %d", pid);
        /* Tell dtrace to exit once truss starts */
        exit(0);
}

This same approach works for attaching a debugger.

# dtrace -wn 'syscall:::entry / execname == "zhyve" / {stop(); system("mdb -p %d", pid); exit(0);}'
dtrace: description 'syscall:::entry ' matched 235 probes
dtrace: allowing destructive actions
CPU     ID                    FUNCTION:NAME
  0    795                 systeminfo:entry Loading modules: [ ld.so.1 ]
>

From the stack trace we can see that we caught it very early. This is even before main() starts.

> $C
ffffbf7fffdff970 ld.so.1`sysinfo+0xa()
ffffbf7fffdffcb0 ld.so.1`setup+0xebd(ffffbf7fffdffe78, ffffbf7fffdffe80, 0, ffffbf7fffdfffd0, 1000, ffffbf7fef3a99c8)
ffffbf7fffdffdd0 ld.so.1`_setup+0x282(ffffbf7fffdffde0, 190)
ffffbf7fffdffe60 ld.so.1`_rt_boot+0x6c()
0000000000000001 0xffffbf7fffdfffc0()

This example was taken from some work I am doing to bring bhyve to SmartOS. In this case, zhyve is the replacement for the init process in a bhyve branded zone.

ksh stack trace

2011-01-03T01:57:23+00:00

It is often times handy to get a stack trace to help you understand how a program got to an error condition. Unfortunately, shell scripting languages tend to not provide an easy mechanism to get a stacktrace. The following examines how it can be accomplished with ksh93.

#! /bin/ksh

function backtrace {
        typeset -a stack 
        # Use "set -u" and an undefined variable access in a subshell
        # to figure out how we got here.  Each token of the result is
        # stored as an element in an indexed array named "stack".
        set -A stack $(exec 2>&1; set -u; unset __unset__; echo $__unset__)

        # Trim the last entries in stack array until we find the one that
        # matches the name of this function.
        typeset i=0
        for (( i = ${#stack[@]} - 1; i >= 0; i-- )); do
                [[ "${stack[i]}" == "${.sh.fun}:" ]] && break
        done

        # Print the name of the function that called this one, stripping off
        # the [lineno] and appending any arguments provided to this function.
        print -u2 "${stack[i-1]/\[[0-9]*\]} $*"
        # Print the backtrace.
        for (( i--; i >= 0; i-- )); do
                print -u2 "\t${stack[i]%:}"
        done
}

# A couple functions to illustrate the output
function a {
        b "$@"
}

function b {
        c "$@"
}

function c {
        # Trigger a backtrace and exit if no arguments were passed
        (( $# == 0 )) && backtrace "missing arg" && exit 1
        print -- "$@"
}

a "$@"

A couple example runs:

$ ./backtrace.ksh hello world!
hello world!
$ ./backtrace.ksh
c: missing arg
        c[37]
        b[32]
        a[28]
        ./backtrace.ksh[41]

This post first appeared on a legacy platform