Skip to content

Virtual Machine Types

Introduction

This document describes the two virtual machine (VM) types minimega is capable of launching - QEMU/KVM virtual machines and containers. minimega uses a common API to describe features and supports launching experiments consisting of both types of VM.

A quick example

Launching a VM is as simple as describing features of the VM, and then instructing minimega to launch one or more VMs based on the current description.

For example, to launch a QEMU/KVM-based VM, we need to specify at least a disk image (or kernel/initrd pair). Say we have a disk foo.qcow2:

example1.mm
# set a disk image
vm config disk foo.qcow2

# set some other common parameters
vm config memory 4096
vm config net 100

# launch one VM, named foo
vm launch kvm foo

# launch 10 more, named bar1, bar2, bar3...
vm launch kvm bar[1-10]

Download this example

Notice we configure the disk image once, but end up launching 11 VMs that use foo.qcow2. minimega always uses the current configuration when launching VMs, which allows you to launch multiple copies of a VM easily.

Also notice that in the vm launch command, we specify that we want to launch a kvm type VM. minimega currently supports the kvm type and the container type, described in detail below.

Common configuration

All VM types are configured using the vm config API. Many configuration parameters are common to all VM types, such as memory, net, and uuid. Others are specific to one type of VM, such as disk, which is specific for KVM type VMs, and filesystem, which is specific to container type VMs.

When launching a VM, only the configuration parameters used by that VM type are used. For example, it is safe to have vm config disk specified when launching a container type VM because the container type doesn't use that configuration parameter.

VM types

KVM Virtual Machines

minimega supports booting kvm type VMs by using QEMU/KVM. When launching kvm type VMs, minimega will use the configuration provided in the vm config API to generate command line arguments for QEMU. Additional configuration, such as creating network taps and assigning them to openvswitch bridges occurs at launch time.

minimega manages running kvm type VMs, including providing an interface to the VM's QMP socket. When a VM quits or crashes, minimega will reflect this in the state column of vm info.

KVM-specific configuration parameters

Most vm config parameters are common to all VM types, though some are specific to KVM and container instances. See the minimega API for documentation on specific configuration parameters.

Configuration parameters specific to KVM instances:

  • vcpus - Set the number of virtual CPUs to allocate for a VM.
  • append - Add an append string to a kernel set with vm kernel.
  • qemu - Set the QEMU process to invoke.
  • qemu-override - Override parts of the QEMU launch string by supplying a string to match, and a replacement string.
  • qemu-append - Add additional arguments to be passed to the QEMU instance.
  • migrate - Assign a migration image, generated by a previously saved VM to boot with.
  • disk - Attach one or more disks to a vm.
  • cdrom - Attach a cdrom to a VM.
  • cpu - Set the virtual CPU architecture.
  • kernel - Attach a kernel image to a VM.
  • initrd - Attach an initrd image to a VM.
  • baremetal - Suppress PC-oriented devices and launch a firmware image with QMP, serial, and tap-network lifecycle support.
  • baremetal-network-driver - Admit a board-integrated QEMU NIC model for bare-metal network configuration.
  • serial - Specify serial ports.
  • virtio-serial - Specify virtio-serial ports.

Bare-metal firmware

The baremetal configuration supports firmware and RTOS guests whose QEMU machine does not provide PC hardware. It removes minimega's implicit VNC, VGA, USB, CD-ROM, keyboard, RTC, PCI bridge, and virtio backchannel devices. QMP lifecycle control, PID tracking, kernel loading, serial sockets, and tap networking remain available.

Bare-metal VMs require vm config kernel, must disable backchannel, and do not support disks, CD-ROMs, migrated state, TPM devices, virtio serial ports, or bidirectional copy and paste. Serial ports use the target board's native UARTs and are exposed as unix sockets in the VM instance directory. Network interfaces use QEMU's board-oriented -net nic,model=... form rather than a PCI bridge, so the configured driver must be implemented by both the selected QEMU machine and the firmware. Set vm config baremetal-network-driver when using a board-integrated controller because QEMU does not report those controllers through its PCI device-discovery interface.

For example, an Arm Cortex-M3 firmware built for QEMU's MPS2 AN385 board can be configured with:

vm config qemu /usr/bin/qemu-system-arm
vm config machine mps2-an385
vm config cpu cortex-m3
vm config memory 16
vm config vcpus 1
vm config kernel /absolute/path/RTOSDemo.out
vm config baremetal true
vm config baremetal-network-driver lan9118
vm config backchannel false
vm config serial-ports 1
vm config net 378,52:54:00:12:34:ad,lan9118
vm launch kvm freertos-mps2
vm start freertos-mps2

The firmware's first UART is then available at serial0 in the VM instance directory. Bare-metal firmware normally has no framebuffer, so VNC is not created and the vnc_port field is zero. QEMU stdout and stderr diagnostics are captured in qemu.log in the same instance directory. Firmware console and Arm semihosting output remain available through the configured serial socket. Firmware that uses Arm semihosting can enable it with:

vm config qemu-append -semihosting -semihosting-config enable=on,target=native

Technical details

minimega uses QEMU version 1.6 or greater. When launching kvm type VMs, the following occurs, in order:

  • A new VM handler is created within minimega, which is populated with a copy of the VM configuration
  • An instance directory for the VM is created (by default /tmp/minimega/N, where N is the VM ID), and the configuration is written to disk
  • If this is a new VM, checks are performed to ensure there are no networking or disk conflicts
  • Any specified network taps are created and attached to openvswitch
  • QEMU arguments are created and patched with any fields specified with vm config qemu-override
  • QEMU is started
  • CPU affinity is applied if enabled
  • Handlers are created to watch the state of QEMU and kill QEMU on demand
  • A QMP connection is established, and QMP capabilities are enabled
  • After a successful QMP connection, the VM is put into the BUILDING state, otherwise QEMU is killed and the VM is put in the ERROR state
  • minimega returns control to the user

A number of QEMU arguments are hardcoded, such as using the host CPU architecture (for kvm support), and enabling memory ballooning. In rare circumstances some of these arguments need to by removed or modified. The vm config qemu-override API provides a mechanism to patch the QEMU argument string before launching VMs.

Containers

minimega supports booting container type VMs via a custom container implementation. A container requires at minimum a root filesystem (rootfs) and an executable to be run at init (PID 1) within the rootfs. The init program can be a shell script. minimega's container implementation supports full-system containers only. This means that every container obtains a PID, network, mount, and IPC namespace. The rootfs must have certain directories populated in order to function correctly. The minimega distribution includes scripts to build busybox-based rootfs (see misc/uminiccc/build.bash and misc/uminirouter/build.bash) and vmbetter-based rootfs (see misc/vmbetter_configs/miniccc_container.conf and misc/vmbetter_configs/minirouter_container.conf).

container-specific configuration parameters

Most vm config parameters are common to all VM types, though some are specific to KVM and container instances. See the minimega API for documentation on specific configuration parameters.

Configuration parameters specific to container instances:

  • hostname - Set a hostname for a container before launching the init program.
  • init - Set the init program and args to exec into upon container launch.
  • preinit - Run processes with elevated privileges before calling init.
  • filesystem - Set the filesystem to use for launching a container.
  • fifo - Set the number of named pipes to include in the container for container-host communication.

Technical details

minimega uses a custom container implementation to boot container type VMs. minimega requires that cgroups be enabled (see notes below for special constraints), and that the linux host support overlayfs. overlayfs is enabled by default in linux 3.18+.

When launching container type VMs, the following occurs, in order:

  • A new VM handler is created within minimega, which is populated with a copy of the VM configuration
  • If this is the first container to be launched, minimega cgroup subtrees will be created for the freezer, memory, and devices cgroup subsystems.
  • If this is a new container, checks are performed to ensure there are no networking or disk conflicts
  • An instance directory and container configuration are written to disk (by default /tmp/minimega/N, where N is the VM ID)
  • If the VM is in snapshot mode, an overlayfs mountpoint is created within the instance directory
  • pipes are created as stdio for the container, as well as container communication that occurs before the container enters the init process
  • A container shim (a copy of minimega with special arguments) is launched under a new set of namespaces
  • minimega creates veth network pairs in the new network namespace of the shim, and connects the host-side device to openvswitch
  • minimega synchronizes with the shim, puts the shim into a paused state (FREEZER in linux cgroup terms), and puts the VM in the building state

When the container shim is launched in a new set of namespaces, the following occurs:

  • Logging is enabled on a special pipe cloned into the process. Logging appears on the parent's log infrastructure (see the log API)
  • stdio is updated with the pipes provided to the shim
  • Specific launch arguments (memory, instance path, etc.) are parsed from the shim command line
  • The hostname is set
  • The rootfs is setup, including default mountpoints (dev, pts, sysfs, proc)
  • /dev is populated and pseudoterminals are created
  • Various symlinks and file masks are applied to prevent the container from having root access to the host
  • The UUID is applied via a special bind mount in sysfs
  • cgroups are setup within the new mount namespace, applying memory and other provided restrictions
  • The container chroots into the rootfs (or overlayfs mount if snapshot mode is enabled)
  • root capabilities are applied to prevent the container from having root access to the host
  • The shim synchronizes with the parent (see above), and the parent minimega puts the container in the FREEZER state
  • When the container is "thawed", the provided init process is started with exec()

Notes

Many linux distributions explicitly disable the memory cgroup, which is required for minimega to boot container type VMs. On debian based linux hosts (including ubuntu), add

cgroup_enable=memory

to the kernel boot parameters (in GRUB or otherwise) to enable the memory cgroup. To enable this in GRUB on Debian, open /etc/default/grub and edit the GRUB_CMDLINE_LINUX_DEFAULT line to include the cgroup parameter. Then run update-grub to update the GRUB config. When you reboot, the cgroup will be enabled.

To better work with systemd-based systems, minimega requires the cgroup hierarchy be mounted as individual mounts (as opposed to one large cgroup mount with all enabled subsystems). If you don't already have a cgroup hierarchy mounted, the following will create a minimal one for minimega:

mount -t tmpfs cgroup /sys/fs/cgroup
mkdir /sys/fs/cgroup/memory
mkdir /sys/fs/cgroup/freezer
mkdir /sys/fs/cgroup/devices
mount -t cgroup cgroup -o memory /sys/fs/cgroup/memory
mount -t cgroup cgroup -o freezer /sys/fs/cgroup/freezer
mount -t cgroup cgroup -o devices /sys/fs/cgroup/devices