FreeBSD as a Host with bhyve
1. Prepare the Host
1.1. 1. Check processor support for bhyve module
Check if the processor supports bhyve is to run dmesg or look in /var/run/dmesg.boot for the POPCNT processor feature flag on the Features2 line for AMD® processors or EPT and UG on the VT-x line for Intel® processors.
root@freebsd:~ # dmesg | egrep 'EPT|UG' --color VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID root@freebsd:~ #
1.2. 2. Load the module
added following lines to /boot/loader.conf on FreeBSD host
# needed for virtualization support vmm_load="YES"
to load the kernel module without rebooting issue the following command
root@freebsd:~ # kldload vmm
1.3. 3. Create bhyve directory structure
For ZFS
zfs create zroot/bhyve
2. Configure bhyve Networking
2.1. NAT (Option 1)
Create single bridge and add a separate tap device for each Guest VMs:
root@freebsd:~ # ifconfig bridge create name brnat up brnat root@freebsd:~ # ifconfig tap create up tap0 root@freebsd:~ # ifconfig brnat addm tap0 root@freebsd:~ # ifconfig brnat inet 10.0.0.1/24
added to /etc/pf.conf
virt_net="10.0.0.0/24" set skip on brnat nat on $ext_if from $virt_net to any -> ($ext_if)
Enable ip forwarding
root@freebsd:~ # sysctl net.inet.ip.forwarding net.inet.ip.forwarding: 1 root@freebsd:~ #
Assign ip to bridge
root@freebsd:~ # ifconfig brnat inet 10.0.0.1/24
An IP can then be assigned to the Guest VM from the 10.0.0.0/24 network and the default gw set to 10.0.0.1.
2.2. Passthrough (Option 2)
FreeBSD Wiki - bhyve PCI Passthrough
root@freebsd:/zroot/bhyve # pciconf -lv vmx1
vmx1@pci0:19:0:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x15ad device=0x07b0 subvendor=0x15ad subdevice=0x07b0
vendor = 'VMware'
device = 'VMXNET3 Ethernet Controller'
class = network
subclass = ethernet
root@freebsd:/zroot/bhyve #
add following to /boot/loader.conf and reboot
pptdevs="19/0/0"
add to bhyve config
bhyve -A -H -P \ -s 0:0,hostbridge \ -s 1:0,lpc \ -s 6,passthru,19/0/0 \ -s 3:0,virtio-blk,./linux.img \ -l com1,stdio \ -c 4 \ -m 2048M linux
3. Create Guest VMs
3.1. Create ZFS volume or disk image
Create a ZFS volume
# zfs create -V16G -o volmode=dev zroot/linuxdisk0
or a disk image with truncate
root@freebsd:~ # truncate -s 16G linux.img
3.1.1. For UEFI based VMs (Option 1)
Install sysutils/bhyve-firmware
Create a per-guest-copy of the variables template file:
# cp /usr/local/share/uefi-firmware/BHYVE_UEFI_VARS.fd /path/to/vm-image/BHYVE_UEFI_VARS.fd
Create a bhyve boot script:
cat start-linux.sh
bhyve -AHP \
-s 0:0,hostbridge \
-s 1:0,lpc \
-s 2:0,virtio-net,tap1 \
-s 3:0,virtio-blk,/dev/zvol/zroot/rheldisc0 \
-s 4:0,ahci-cd,./rhel-8.10-x86_64-dvd.iso \
-c 4 \
-m 2048M \
-u \
-s 29,fbuf,password="PASSWORDHERE",tcp=0.0.0.0:5900 \
-s 30,xhci,tablet \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,./BHYVE_UEFI_VARS.fd \
rhel
Start the VM.
# ./start-linux.sh
Stop the VM when needed:
# cat ./destroy.sh bhyvectl --destroy --vm=debian1
3.1.2. Booting with GRUB (Option 2)
Install sysutils/grub2-bhyve port.
Create a device.map that grub will use to map the virtual devices to the files on the host system:
root@freebsd:/zroot/bhyve # cat device.map (hd0) ./linux.img (cd0) ./ubuntu-22.04.4-live-server-amd64.iso root@freebsd:/zroot/bhyve #
- Use sysutils/grub2-bhyve to load the Linux® kernel from the ISO image:
root@freebsd:/zroot/bhyve # grub-bhyve -m device.map -r cd0 -M 1024M linux
- Now that the Linux® kernel is loaded, the guest can be started:
root@freebsd:/zroot/bhyve # bhyve -A -H -P \
-s 0:0,hostbridge \
-s 1:0,lpc \
-s 2:0,virtio-net,tap0 \
-s 3:0,virtio-blk,./linux.img \
-s 4:0,ahci-cd,./ubuntu-22.04.4-live-server-amd64.iso \
-l com1,stdio \
-c 4 \
-m 1024M linux
- The system will boot and start the installer. After installing a system in the virtual machine, reboot the virtual machine. This will cause bhyve to exit. The instance of the virtual machine needs to be destroyed before it can be started again:
root@freebsd:~ # bhyvectl --destroy --vm=linux
4 (a). Now the guest can be started directly from the virtual disk. Load the kernel:
grub> ls (hd0) (hd0,gpt3) (hd0,gpt2) (hd0,gpt1) (cd0) (cd0,gpt3) (cd0,gpt2) (cd0,gpt1) (host) (lvm/ubuntu--vg-ubuntu--lv) grub> ls (hd0,gpt2)/ lost+found/ grub/ config-5.15.0-94-generic initrd.img.old vmlinuz.old System.map-5.15.0-94-generic vmlinuz-5.15.0-94-generic initrd.img vmlinuz initrd.img-5.15.0-94-generic grub> linux (hd0,gpt2)/vmlinuz-5.15.0-94-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv grub> initrd (hd0,gpt2)/initrd.img-5.15.0-94-generic grub> boot
4 (b). Boot the new system
root@freebsd:/zroot/bhyve # cat boot.sh
bhyve -A -H -P \
-s 0:0,hostbridge \
-s 1:0,lpc \
-s 2:0,virtio-net,tap0 \
-s 3:0,virtio-blk,./linux.img \
-l com1,stdio \
-c 4 \
-m 1024M \
-u \
linux
root@freebsd:/zroot/bhyve #
When finished reboot and destroy:
guestvm# reboot bhyvehost# bhyvectl --destroy --vm=debian
- Create a start script which automatically loads the kernel (4 a.) and boots the VM (4 b.):
#!/bin/sh
grub-bhyve -m device.map -r hd0,gpt2 -M 2048M linux < linux-grub.cfg > /dev/null
bhyve -A -H -P \
-s 0:0,hostbridge \
-s 1:0,lpc \
-s 2:0,virtio-net,tap0 \
-s 3:0,virtio-blk,./linux.img \
-l com1,stdio \
-c 8 \
-m 2048M \
-u \
linux
device.map
(hd0) ./linux.img (cd0) ./ubuntu-22.04.4-live-server-amd64.iso
linux-grub.cfg
linux (hd0,gpt2)/vmlinuz-5.15.0-122-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv initrd (hd0,gpt2)/initrd.img-5.15.0-122-generic boot
4. Start VMs when FreeBSD Host boots
crontab
root@freebsd:~ # crontab -l @reboot /zroot/bhyve/linux-nat.sh @reboot cd /zroot/bhyve/ubuntu; /usr/local/bin/tmux new-session -d -s "ubuntu" /zroot/bhyve/ubuntu/linux-start.sh root@freebsd:~ #
/zroot/bhyve/linux-nat.sh
root@freebsd:~ # cat /zroot/bhyve/linux-nat.sh #!/bin/sh ifconfig bridge create name brnat up ifconfig tap create up ifconfig tap create up ifconfig brnat addm tap0 ifconfig brnat addm tap1 ifconfig brnat inet 10.0.0.1/24 root@freebsd:~ #