vm-bhyve/vm.8

.Dd Jun 30, 2015
.Dt VM-BHYVE 8
.Os
.Sh NAME
.Nm vm
.Nd "utility to manage bhyve virtual machines"
.Sh SYNOPSIS
.Nm
.Cm init
.Pp
.Nm
.Cm switch list
.Nm
.Cm switch create
.Ar name
.Nm
.Cm switch vlan
.Ar name vlan-id
.Nm
.Cm switch nat
.Ar name on|off
.Nm
.Cm switch add
.Ar name interface
.Nm
.Cm switch remove
.Ar name interface
.Nm
.Cm switch destroy
.Ar name
.Pp
.Nm
.Cm create
.Op Fl t Ar template
.Op Fl s Ar size
.Ar name
.Nm
.Cm destroy
.Ar name
.Nm
.Cm list
.Nm
.Cm install
.Ar name iso
.Nm
.Cm start
.Ar name
.Nm
.Cm stop
.Ar name
.Nm
.Cm console
.Ar name
.Nm
.Cm reset
.Ar name
.Nm
.Cm poweroff
.Ar name
.Nm
.Cm startall
.Nm
.Cm stopall
.Nm
.Cm configure
.Ar name
.Pp
.Nm
.Cm iso
.Op Ar url
.Pp
.Nm
.Cm image list 
.Nm
.Cm image create
.Op Fl d Ar description
.Ar name
.Nm
.Cm image provision
.Ar uuid
.Ar new-name
.Nm
.Cm image destroy
.Ar uuid
.\" ============ DESCRIPTION =============
.Sh DESCRIPTION
The
.Nm
utility is used to provide simplified management of bhyve virtual machines,
including networking and console access.
.Pp
Networking is handled by creating one or more virtual switches. Each switch
has a simple name which is referenced in the virtual machine configuration file.
The
.Nm
utility automatically creates a
.Xr bridge 4
device for each virtual switch and assigns virtual machine
.Xr tap 4
interfaces dynamically.
.Pp
All configuration for virtual machines is stored in a simple rc style configuration
file. When virtual machines are first created, the configuration file is copied from
a template which can be specified by the user. Multiple templates can be created providing
an easy way to provision guests with specific configurations.
.Pp
.Nm
gracefully handles reboot and shutdown commands from inside the guests, whilst providing
full management of the virtual machine from the host system.
.\" ============ BASIC SETUP ============
.Sh BASIC SETUP
Once
.Nm
is installed, create the directory which will store your virtual machine configuration and data.
This directory will be referred to as
.Pa $vm_dir
throughput this man page.
.Pp
Add the following into
.Pa /etc/rc.conf
.Bd -literal -offset indent
$vm_enable="YES"
$vm_dir="/your/vm/path"
$vm_list=""
$vm_delay="5"
.Ed
.Pp
The first and second lines are required to enable the
.Nm
utiltity. Please see the
.Cm startall
command description for more information on the third and fourth settings.
.Pp
Now run the
.Nm vm
.Cm init
command to finish initialisation. This will create subdirectories inside
.Pa $vm_dir
to hold configuration and templates. It will also load any required kernel modules.
This command needs to be run on each boot, which is normally handled by the rc.d script.
.Pp
Copy the sample templates into the directory
.Pa $vm_dir/.templates/ .
You can create and edit the templates as required. It is recommended to keep a template called
.Pa default.conf ,
as this will be used when no template is manually specified.
.\" ============ ZFS =============
.Sh ZFS
If you are using a ZFS dataset to store your virtual machines, and want a new child dataset created
for each one, specify the dataset to use in
.Pa /etc/rc.conf
as follows:
.Bd -literal -offset indent
$vm_dir="zfs:pool/dataset"
.Ed
.Pp
In contrast to earlier versions, if
.Pa $vm_dir
is a normal path, a standard subdirectory will be created for each virtual machine, regardless
of the file system type. However,
.Nm
is now able to handle situations where the dataset mountpoint does not match the dataset name.
.\" ============ QUICKSTART =============
.Sh QUICKSTART
Create a virtual switch called
.Sy public
(which is the switch name specified in the default templates) and attach it to a real interface.
Use your own interface in place of
.Sy em0
as required.
.Bd -literal -offset ident
# vm switch create public
# vm switch add public em0
.Ed
.Pp
Download an ISO file to use for installation:
.Bd -literal -offset ident
# vm iso ftp://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/10.1/FreeBSD-10.1-RELEASE-amd64-disc1.iso
.Ed
.Pp
Create a new guest using the default template and disk size, then start the installation. The
.Ar install
subcommand will pause while the guest boot loader completes. Once successfully loaded, use the
.Ar console
command to complete the installation.
.Bd -literal -offset ident
# vm create my-guest
# vm install my-guest FreeBSD-10.1-RELEASE-amd64-disc1.iso
# vm console my-guest
.Ed
.Pp
Please note that Linux guests currently require the
.Sy sysutils/grub2-bhyve
package to be installed. This is use in place of
.Xr bhyveload 8
to load the guest kernel into memory.
.\" ============ SUBCOMMANDS =============
.Sh SUBCOMMANDS
.Bl -tag -width indent
.It Cm init
.br
This should be run once after each host reboot before running any other
.Nm
commands. The main function of the
.Cm init
command is as follows:
.Pp
o Load all necessary kernel modules if not already loaded
.br
o Set tap devices to come up automatically when opened
.br
o Create any configured virtual switches
.It Cm switch list
List virtual switches. This reads all configured virtual switches from the
.Pa $vm_dir/.config/switch
file and displays them. If the virtual switches are loaded, it also tries
to display the
.Xr bridge 4
interface that has been assigned to each one.
.It Cm switch create Ar name
Create a new virtual switch. The name must be supplied and may only contain
letters, numbers and dashes. However, it may not contain a dash at the beginning
or end.
.Pp
When a new virtual switch is created, the persistent configuration file is updated
and a new
.Xr bridge 4
interface is provisioned.
.It Cm switch vlan Ar name Ar vlan-id
Assign a VLAN number to a virtual switch. The VLAN number must be between 0-4094.
.Pp
When adding an interface to a VLAN enabled virtual switch, a new
.Xr vlan 4
interface is created. This interface has the relevent parent interface and VLAN tag
configured. This vlan interface is then added to the virtual switch. As such, all
traffic between guests on the same switch is untagged and travels freely. However,
all traffic exiting via physical interfaces is tagged.
.Pp
If the virtual switch already has physical interfaces assigned, they are all removed
from the bridge, reconfigured, then re-added.
.Pp
To remove the VLAN configuration from a virtual switch, specify a
.Ar vlan-id
of 0.
.It Cm switch nat Ar name Ar on|off
Enable or disable NAT functionality on the specified switch. Both
.Xr pf
and
.Xr dnsmasq
are required for this function to be available.
.Pp
The switch should have no host ports assigned, as these will end up on the private side
of the NAT network.
.Nm
automatically detects the hosts default gateway, which is used as the forwarding interface
for NAT connections.
.Pp
Once enabled, a 172.16.X.0/24 network is assigned to the switch (bridge) interface.
.Ar X
is chosen based on the ID of the bridge interface. For example, if the switch is using
bridge10, the network will be 172.16.10.0/24.
.Xr dnsmasq
is used to provide DHCP to the guests, and
.Xr pf
rules are inserted to provide the NAT translation.
.Pp
Please note that
.Nm
will overwrite the dnsmasq configuration file
.Pa /usr/local/etc/dnsmasq.conf
as it creates a specific configuration to provide DHCP on each switch interface.
.Pa /etc/pf.conf
is created if it doesn't exist, and a single include statement is added. This
include statement can be moved within the file if required.
.It Cm switch add Ar name Ar interface
Add the specified interface to the named virtual switch.
.Pp
The interface will immediately be added to the relevant bridge if possible, and
stored in the persistent switch configuration file. If a
.Ar vlan-id
is specified on the virtual switch, this will cause a new
.Xr vlan 4
interface to be created.
.It Cm switch remove Ar name Ar interface
Removes the specified interface from the named virtual switch and updates the
persistent configuration file.
.It Cm switch destroy Ar name
Completely remove the named virtual switch and all configuration. The associated
.Xr bridge 4
interface will be removed, as well as any
.Xr vlan 4
interfaces if they are not in use by other virtual switches.
.It Xo
.Cm create
.Op Fl t Ar template
.Op Fl s Ar size
.Ar name
.Xc
Create a new virtual machine.
.Pp
Unless specified, the
.Pa default.conf
template will be used and a 20GB virtual disk image is created. This command will
created the virtual machine directory
.Pa $vm_dir/$name ,
and create the configuration file and empty disk image within.
.Bl -tag -width 12n
.It Fl t Ar template
Specifies the template to use from within the
.Pa $vm_dir/.templates
directory. The
.Sy .conf
suffix is not required.
.It Fl s Ar size
The size of disk image to create in GB. Unless specified, the guest image will
be a sparse file 20GB in size.
.El
.It Cm destroy Ar name
Removes the specified virtual machine from the system, deleting all associated
disk images & configuration.
.It Cm list
.br
List all the virtual machines in the
.Pa $vm_dir/.templates
directory. This will show the basic configuration for each virtual machine, and whether
they are currently running.
.It Cm install Ar name Ar iso
Start a guest installation for the named virtual machine, using the specified ISO file.
The
.Ar iso
argument should be the filename of an ISO file already downloaded into the
.Pa $vm_dir/.iso
directory. ISO files in this directory can be managed using the
.Ar iso
subcommand described below.
.Pp
Once started, the guest loader will be booted in the foreground. This allows you to choose
the
.Sy Install
boot option for guests that require it. Once the loader has completed, you will be returned
to the shell and bhyve will continue running in the background. Use the
.Ar console
subcommand to connect to the guest and complete installation.
.Pp
After installation, the guest can be rebooted and will restart using its own disk image to boot.
At this point the installation ISO file is still attached, allowing you to use the CD/DVD image
for any post installation tasks. The ISO file will remain attached after each reboot until the
guest is fully stopped.
.It Cm start Ar name
Start the named virtual machine. The guest will boot and run completely in the background. Use
the
.Ar console
subcommand to connect to it if required.
.Pp
For each network adapter specified in the guest configuration, a
.Xr tap 4
interface will be created. If possible, the tap interface will be attached the relevent
.Xr bridge 4
interface, based on the virtual switch specified in the guest configuration.
.It Cm stop Ar name
Stop a named virtual machine. All
.Xr tap 4
and
.Xr nmdm 4
devices will be automatically cleaned up once the guest has exited.
.It Cm console Ar name
Connect to the console of the named virtual machine. Without network access, this is the primary
way of connecting to the guest once it is running.
.Pp
This looks for the
.Xr nmdm 4
device associated with the virtual machine, and connects to it with
.Xr cu 1 .
Use ~+Ctrl-D to exit the console and return to the host.
.It Cm reset Ar name
Forcefully reset the named virtual machine. This can cause corruption to the guest file system just
as with real hardware and should only be used if necessary.
.It Cm poweroff Ar name
Forcefully power off the named virtual machine. As with
.Ar reset
above, this does not inform the guest to shutdown gracefully and should only be used if the guest
can not be shut down using normal methods.
.It Cm startall
Start all virtual machines configured for auto-start. This is the command used by the rc.d scripts
to start all machines on boot.
.Pp
The list of virtual machines should be specified using the
.Pa $vm_list
variable in
.Pa /etc/rc.conf .
This allows you to use shared storage for virtual machine data, whilst making sure that the correct
guests are started automatically on each host. (Or to just make sure your required guests start on boot
whilst leaving test/un-needed guests alone)
.Pp
The delay between starting guests can be set using the
.Pa $vm_delay
variable, which defaults to 5 seconds. Too small a delay can cause problems as each guest doesn't
have enough time to claim a null modem device before the next guest starts. Increasing this value
can be useful if you have disk-intensive guests and want to give each guest a chance to fully
boot before the next starts.
.It Cm stopall
Stop all running virtual machines. This sends a stop command to all
.Xr bhyve 8
instances, regardless of whether they were starting using
.Nm
or not.
.It Cm configure Ar name
The
.Cm configure
command simply opens the virtual machine configuration file in your default editor,
allowing you to easily make changes. Please note, changes do not take effect until
the virtual machine is fully shutdown and restarted.
.It Cm iso Op Ar url
List all the ISO files currently stored in the
.Pa $vm_dir/.iso
directory. This is often useful during guest installation, allowing you to copy and paste the ISO
filename.
.Pp
If a
.Sy url
is specified, instead of listing ISO files, it attempts to download the given file using
.Xr fetch 1 .
.It Cm image list
List available images. Any virtual machine can be packaged into an image, which can then be
used to create additional machines. All images have a globally unique ID (UUID) which is 
used to identify them. The list command shows the UUID, the original machine name, the
date it was created and a short description of the image.
.Pp
Please note that these commands rely on using ZFS featured to package/unpackage the images,
and as such are only available when using a ZFS dataset as the storage location.
.It Xo
.Cm image create
.Op Fl d Ar description
.Ar name
.Xc
Create a new image from the named virtual machine. This will create a compressed copy of
the original guest dataset, which is stored in the
.Pa $vm_dir/images
directory. It also creates a 
.Pa UUID.manifest
file which contains details about the image.
.Pp
Once complete, it will display the UUID which has been assigned to this image.
.It Cm image provision Ar uuid Ar new-name
Create a new virtual machine, named
.Pa new-name ,
from the specified image UUID.
.It Cm image destroy Ar uuid
Destroy the specified image.
.El
.\" ============ CONFIGURATION FORMAT ===========
.Sh CONFIGURATION FORMAT
Each virtual machine has a configuration file that specifies the hardware configuration. This
uses a similar format to the
.Sy rc
files, making them easy to edit by hand. The settings for each guest are stored in
.Pa $vm_dir/$vm_name/$vm_name.conf .
An overview of the available configuration options is listed below.
.Bl -tag -width 15n
.It guest
Specify the type of guest to be installed in this virtual machine. Current valid options
for this are freebsd, netbsd, openbsd centos, ubuntu, debian & alpine linux.
.It cpu
A numeric value specifying the number of virtual CPU cores to assign to the guest.
.It memory
The amount of memory to assign to the guest. This can be specified in megabytes or
gigabytes using the
.Sy M
and
.Sy G
suffixes.
.It network0_type
The emulation to use for the first network adapter. This option can be unspecified
if no guest networking is required. The recommended value for this is
.Sy virtio-net .
Additional network interfaces can be configured by adding additional
.Sy networkX_type
and
.Sy networkX_switch
values, replacing
.Sy X
with the next available integer.
.It network0_switch
The virtual switch to connect interface
.Sy 0
to. This should correspond to a virtual switch created using the
.Pa vm switch create
subcommand. If the virtual switch is not found, an interface will still be assigned,
but not connected to any bridge.
.It disk0_type
The emulation type for the first virtual disk. At least one virtual disk is required.
Valid options for this are currently
.Sy virtio-blk
and
.Sy ahci-hd .
Additional disks can be added by adding additional
.Sy diskX_type
and
.Sy diskX_name
values, replacing
.Sy X
with the next available integer.
.It disk0_name
The filename for the first virtual disk. The first disk is created automatically when
provisioning a new virtual machine. If additional disks are added, the image will need
to be manually created, usually done using the
.Xr truncate 1
command.
.It disk0_dev
The type of device to use for the disk. If not specified, this will default to 'file',
and a sparse file, located in the guest directory, will be used as the disk image.
Other options include 'zvol' & 'sparse-zvol', which will used a ZVOL as the disk image.
.Pp
The primary disk will be created automatically when the virtual machine is created.
Any additional disks will need to be created manually, using either then
.Xr truncate 1
or
.Xr zfs 8
command, depending on the type of disk device required.
.It passthruX
Specify a device to pass through to the guest. You will need to reserve the device first
so that is it claimed by the ppt driver on boot.
.Pp
Once the device is successfully reserved, you can add it to the guest by adding
.Sy passthruX="1/2/3"
to the guest configuration file, where
.Sy X
is an integer starting at 0, and
.Sy 1/2/3
is the Base/Slot/Function of the device. If you are passing through multiple functions on
the same device, make sure they are specified together in the configuration file in the
same sequence as the original device.
.Pp
Please see https://wiki.freebsd.org/bhyve/pci_passthru for more details on how this works.
.It linux_kernel
CentOS guests require the kernel to be loaded inside the boot loader. This option is
required for these guests and specifies the kernel version number to load. Hopefully
this requirement will be relaxed as
.Xr bhyve 8
development continues.
.El
.\" ============ SEE ALSO =============
.Sh SEE ALSO
.Xr bhyve 8 ,
.Xr bhyveload 8 ,
.Xr tap 4 ,
.Xr bridge 4 ,
.Xr vlan 4 ,
.Xr nmdm 4 ,
.Xr cu 1 ,
.Xr fetch 1 ,
.Xr truncate 1 ,
.Xr zfs 8
.\" ============ BUGS =============
.Sh BUGS
Please report all bugs/issues/feature requests to the github project at
https://github.com/churchers/vm-bhyve
.\" ============ AUTHOR =============
.Sh AUTHOR
.An Matt Churchyard Aq Mt churchers@gmail.com