Xen

Xen is a native, or bare-metal, hypervisor that allows multiple distinct virtual machines (referred to as domains) to share a single physical machine. As the highest privilege process on the system, Xen is responsible for the distribution of processor and memory resources between guest domains on the host. Other hardware resources such as network interfaces, disks, or direct PCI/USB devices are controlled by a privileged domain known as domain-0 (dom0).

From it's inception Xen has focused on the para-virtualization approach to hypervisor design. As a result, Xen guests or unprivileged domains (domU) are typically aware of the hypervisor and their status as guests. The base system, Domain-0, must have inherent Xen support, however, unmodified domU guests are supported on hardware which implements Intel (VT-x) or AMD (SVM) virtualization technology.

BIOS and UEFI firmware
While paravirtualized (PV) guests can run on a host without any form of hardware accelerated virtualization extensions this severely restricts the set of available guests and prevents the use of new and exciting modes such as PVH.

The following command should provide a highlighted row for each processor if the corresponding feature is present:

Unfortunately motherboard manufacturers often explicitly disable these virtualization extensions via BIOS or UEFI settings. If you don't see support listed check your BIOS/UEFI configuration for flags related to virtualization.

Rebuilding the Gentoo installation?
A dramatic change that might be necessary on 32-bit systems is to rebuild the entire Gentoo installation with a different  setting. Guest operating systems running under Xen might otherwise see major performance degradation. If you, however, are planning on checking out Xen rather than installing it for production use and are not terribly fond of rebuilding all programs, you can skip this step. In this case you will notice performance degradation but you will still be able to use Xen.

Add -mno-tls-direct-seg-refs ONLY if you have a 32-bit dom0. You don't need this flag if you have a 64-bit dom0.

If you boot your system using an initial ramdisk (initrd) you need to rebuild the initrd as well (which is best done by running all steps you would do when you rebuild your kernel).

Kernel
Next we'll build the Linux kernel with Xen support. This kernel, whose sources are available at, will be our main running kernel (i.e. the one running domain 0). In the  section you'll find drivers for all kinds of input/output, each driver having a backend and frontend implementation available. For the domain 0 kernel you need to select the backend implementation: these are used by the other domains (who use the frontend drivers) to communicate directly with the hardware. However, you should be able to configure the kernel to provide support for both frontend (guest) and backend (host) drivers.

If you're wondering about networking: each interface in a domain has a point-to-point link to an interface on domain 0 (called where X is the domain number and Y the Yth interface of that domain), so you can configure your network the way you want (bridging, NAT, etc.)

Enable general Xen support:

Add support for paravirtualized console connections:

Facilitates guest access to block and network devices via dom0:

In some configurations it can be desirable to provide a guest with direct access to a PCI device. This is known as Xen PCI Passthrough:

Keyboard, mouse, and display support via dom0 backend:

Xen dom0 support depends on APCI; without it dom0 related options will be hidden:

Typical network configurations depend on linux bridge functionality:

Fully virtualised (HVM) guests depend on Universal TUN/TAP device driver support for their network interfaces. This option is required if you plan to create fully Emulated Network Devices within Dom0/DomU configuration.

The remaining drivers flesh out memory management, domain-to-domain communication, and communication to Xen via sysfs interfaces:

With all of the above configuration enabled, this kernel image should be able to boot as the dom0 host or as another domU guest. Note that the domU kernel can be slimmed down significantly if desired.

USE flags
The first set of use flags correspond directly to the Xen hypervisor. Of these, ensure that EFI support is selected if you plan to configure EFI boot.

In addition to the core hypervisor Xen depends on a set of supporting libraries and management tools.

Emerge
Xen will automatically deploy the compiled, boot-able hypervisor in. If you have a separate boot file system ensure that it is mounted.

After you have reviewed and updated the relevant USE flags, emerge xen and xen-tools:

Once the installation process completes should be present.

Performance tuning
When configuring a host with a significant amount of memory it is beneficial to fix the allocated to dom0 and disable ballooning functionality. This ensures Xen and the dom0 operating system are able to build working structures based on the memory available at boot-time and have their assumptions hold throughout operation. Additionally, in the event that a large number of guests are created, dom0 is explicitly protected from external memory pressure.

In a GRUB2 environment the following directive assigns 1 GiB of memory to the dom0 guest:

Ballooning should also be disabled by setting autoballoon to either "off" or "auto" in.

Further detail on this topic and other related tuning suggestions can be found on the Tuning Xen for Performance article on the Xen wiki.

Runlevel
In order for your new Xen environment to function correctly a number of daemons must be started. The following table provides a brief overview of each daemon and its function.

To add these daemons directly to the default runlevel perform the following:

These daemons should only be started when the system boots into Xen. As this may not always be the case, it may be desirable to keep these services out of the default runlevel. One possible alternative is to utilize an alternate default runlevel via the softlevel kernel parameter. The following example, based on this blog post, creates a new runlevel named xen.

First create the new runlevel:

Populate it with everything currently present in the existing default runlevel:

Then instead of adding the xen daemons to the default runlevel add them to the xen level:

Once all of these steps have been completed add the following statement to the grub default file:

Bootloader
In order for Xen to be the highest privilege process it must be the initial boot target, however, it must also subsequently boot the dom0 kernel.

Legacy GRUB
The following configuration handles basic GRUB settings:

In the above configuration GRUB will load xen.gz as the kernel with the actual dom0 kernel provided as a module.

GRUB2
GRUB2 provides auto-configuration scripts through which will generate the appropriate settings based on the kernel. If the scripts detect Xen dom0 options they will append Xen hypervisor boot lines to the grub menu. Note that for this to function correctly the kernel config file must be located in one of the following directories with a suffix matching the desired kernel:

The example column above assumes a kernel named /boot/kernel-4.0.5-gentoo.

The following variables in are the most common ones to set to control how GRUB2 will function:

For a more complete list, please refer to the GRUB2 configuration variables sub-page.

GRUB2 support is also included in and can be triggered during the build process:

Booting Xen
Once you are satisfied with your configuration, reboot the system into Xen. Assuming everything worked correctly Xen should transfer display control to the dom0 environment, which will operate just like the original bare-metal system.

Running should result in output similar to the following if Xen is functioning correctly:

At this point you may want to configure your system to boot into Xen by default. If you are utilizing GRUB2, add the following lines to the grub defaults file:

Then, copy the name or ID of the Xen boot config and set it via grub2-set-default:

Building the kernel
Go to the Xen-powered Linux kernel source and, if necessary, update the configuration. It is wise to keep as many topics as possible similar to the main kernel. Then build the kernel and place the resulting file where you want (we assume this is  ):

If you'd like to trim down the domU kernel the following flags are necessary.

Enable general Xen support:

Facilitates guest access to block and network devices via dom0:

In some configurations it can be desirable to provide a guest with direct access to a PCI device. This is known as Xen PCI Passthrough:

Keyboard, mouse, and display support via dom0 backend:

The remaining drivers flesh out memory management, domain-to-domain communication, and communication to Xen via sysfs interfaces:

Creating the domain disks
For best performance, it is best to dedicate a partition (or logical volume) to a domain rather than a file based filesystem. However, if you are going to use Xen primarily for tests using a file based filesystem does have its advantages (especially regarding maintenance).

You can create a file based filesystem using and  (or any other file system creation tool). For instance, to create a 4 Gbyte ext4 filesystem:

Configuring a domain
Next we create a Xen configuration file for a domain. You can store these configuration files where you want, for instance at. As an example, we create a configuration file for a small Gentoo environment which uses the disk image we created previously:

If you are using a block device (such as an lvm volume or partition) for the disk use 'phy:' instead of 'file:' and leave off /dev. For example:

You can find example configuration files in, which is also the default location for domU config files.

Launching the new domain
Now we're all set and we can launch the new domain. If the disk image contained an operating system, we could just create and attach the domain using the command:

The domain would be booted inside the terminal in which you executed the command. However, in our case, the disk image is empty so the domain won't boot up in anything useful. To fix this, you can loop-mount the image and install Gentoo as you're used to.

If you want to disconnect from the domain, press +. You can always reconnect to the domains' console using. However, there is only one console per domain, so only use it when you can't access the domain otherwise (for instance, through SSH).

If you are missing login prompt on the console, make sure you have entries like this in your inittab files on dom0 and domU pointing to :

To apply the changes in without a reboot issue the following command:

If it still does not work, check the kernel config for DEVTMPFS and DEVTMPFS_MOUNT. These two values should be enabled in either module or built in format.

Introduction
Xen works best when using a bridged mode network configuration. This means that your default network interface on the administrative domain becomes a bridge which accepts connections to the virtual domains as well as to the IP address your administrative domain has.

Bridged interfaces
Create a bridge interface by creating a new link to the networking init script as provided by Gentoo:

Next, edit and setup the bridge:

Finally, install the package, and make sure the  init script is loaded at boot.

If you use bridged networks with real internet IP's in hosted environments, it may be necessary to add one or all of the following lines (depending on your environment) in your file to prevent redirects, that can cause intermittent network interruptions:

To get the changes in to work, use:

if you encounter poor network performance or if your domU network permanently stops working under heavy load (backup jobs, etc) (from outside it looks like the instance would crash, but deactivating and activating the interface e.g. from the  with , restores normal operation) , use   to improve/prevent it on all interfaces connected to the bridge (don't forget the bridge itself):

External resources

 * Xen Wiki
 * virt-manager - A graphical tool for administering virtual machines.
 * Xen network tuning