Xen

From Gentoo Wiki
Jump to: navigation, search
Resources

Xen is a native, or bare-metal, hypervisor that allows multiple distinct virtual machines (referred to as domains) to share a single physical machine. As the highest privilege process on the system, Xen is responsible for the distribution of processor and memory resources between guest domains on the host. Other hardware resources such as network interfaces, disks, or direct PCI/USB devices are controlled by a privileged domain known as domain-0 (dom0).

From it's inception Xen has focused on the para-virtualization approach to hypervisor design. As a result, Xen guests or unprivileged domains (domU) are typically aware of the hypervisor and their status. The base system, Domain-0, must have inherent Xen support, however, unmodified domU guests are supported on hardware which implements Intel (VT-x) or AMD (SVM) virtualization technology.

Preparing Domain-0 (dom0)

Introduction

Domain0 is the primary domain under Xen, hosting the host operating system which governs all other domains. In this chapter we will prepare an existing Gentoo installation to become the host operating system in this domain and build the Xen-powered kernel so that Gentoo is ready to host other Xen domains.

Rebuilding the Gentoo Installation?

A dramatic change that might be necessary on 32-bit systems is to rebuild the entire Gentoo installation with a different CFLAGS setting. Guest operating systems running under Xen might otherwise see major performance degradation. If you, however, are planning on checking out Xen rather than installing it for production use and are not terribly fond of rebuilding all programs, you can skip this step. In this case you will notice performance degradation but you will still be able to use Xen.

Important
It is advised that, if you change your CFLAGS and build your system with a gcc lower than version 4, you do not have -Os set as it has been reported to produce broken code.

Add -mno-tls-direct-seg-refs ONLY if you have a 32-bit dom0. You don't need this flag if you have a 64-bit dom0.

FILE /etc/portage/make.confCFLAGS change for mno-tls-direct-seg-refs
CFLAGS="... -mno-tls-direct-seg-refs"
root #emerge -e world

If you boot your system using an initial ramdisk (initrd) you need to rebuild the initrd as well (which is best done by running all steps you would do when you rebuild your kernel).

Installing Xen

Xen actually contains many components, so you'll need to install a few packages.

root #emerge xen xen-tools gentoo-sources

Building the Kernel

Next we'll build the Linux kernel with Xen support. This kernel, whose sources are available at /usr/src/linux , will be our main running kernel (i.e. the one running domain 0). In the XEN section you'll find drivers for all kinds of input/output, each driver having a backend and frontend implementation available. For the domain 0 kernel you need to select the backend implementation: these are used by the other domains (who use the frontend drivers) to communicate directly with the hardware. However, you should be able to configure the kernel to provide support for both frontend (guest) and backend (host) drivers.

If you're wondering about networking: each interface in a domain has a point-to-point link to an interface on domain 0 (called vifX.Y where X is the domain number and Y the Yth interface of that domain), so you can configure your network the way you want (bridging, NAT, etc.)

Enable general Xen support:

KERNEL Xen Base
Processor type and features  --->
     [*] Linux guest support  --->
          [*]   Enable paravirtualization code
          [*]     Xen guest support
          [*]       Support for running as a PVH guest
          [*]     Paravirtualization layer for spinlocks

Add support for paravirtualized console connections:

KERNEL PV Console
Device Drivers  --->
     Character devices  --->
          [*] Xen Hypervisor Console support
          [*]   Xen Hypervisor Multiple Consoles support

Facilitates guest access to block and network devices via dom0:

KERNEL Disk and Network
Device Drivers  --->
     [*] Block devices  --->
          <*>   Xen virtual block device support
     [*] Network device support  --->
          <*>   Xen network device frontend driver

In some configurations it can be desirable to provide a guest with direct access to a PCI device. This is known as Xen PCI Passthrough :

KERNEL Guest PCI Passthrough
Bus options (PCI etc.)  --->
     [*] Xen PCI Frontend

Keyboard, mouse, and display support via dom0 backend:

KERNEL Guest Human Interface
Device Drivers  --->
     Input device support  --->
          [*]   Miscellaneous devices  --->
               <*>   Xen virtual keyboard and mouse support
     Graphics support  --->
          Frame buffer Devices  --->
               <*> Xen virtual frame buffer support

Xen dom0 support depends on APCI; without it dom0 related options will be hidden:

KERNEL APCI Support
Power management and ACPI options  --->
     [*] ACPI (Advanced Configuration and Power Interface) Support  --->

Typical network configurations depend on linux bridge functionality:

KERNEL Linux Bridge
[*] Networking support --->
     Networking options  --->
          <*> 802.1d Ethernet Bridging
	  [*] Network packet filtering framework (Netfilter) --->
	       [*] Advanced netfilter configuration
	       [*]   Bridged IP/ARP packets filtering

The remaining drivers flesh out memory management, domain-to-domain communication, and communication to Xen via sysfs interfaces:

KERNEL Xen Drivers
Device Drivers  --->
     [*] Block devices  --->
          <*>   Xen block-device backend driver
     [*] Network device support --->
          <*>   Xen backend network device
     Xen driver support  --->
          [*] Xen memory balloon driver
          [*]   Scrub pages before returning them to system
          <*> Xen /dev/xen/evtchn device
          [*] Backend driver support
          <*> Xen filesystem
          [*]   Create compatibility mount point /proc/xen
          [*] Create xen entries under /sys/hypervisor
          <*> userspace grant access device driver
          <*> User-space grant reference allocator driver
          <M> Xen PCI-device backend driver
          <*> Xen ACPI processor
          [*] Xen platform mcelog

With all of the above configuration enabled, this kernel image should be able to boot as the dom0 host or as another domU guest. Note that the domU kernel can be slimmed down significantly if desired.

Bootloader

Once the kernel is built you'll find the kernel image immediately in the build directory (not inside arch/ or any other directory) called vmlinuz . Copy it to /boot and then configure your bootloader to use the Xen hypervisor (one of the components installed previously) which is stored as /boot/xen.gz . In the bootloader configuration, add your newly built kernel as the kernel that Xen should boot. For instance, for GRUB:

FILE /boot/grub/grub.confGRUB Configuration for Xen
title Xen Gentoo Linux 3.5
root (hd0,0)
kernel /boot/xen.gz
module /boot/kernel-3.5.x.y-xen0 root=/dev/sda3

If you are using grub2, which provides auto-configuration scripts through grub2-mkconfig, you can also copy your kernel .config as config-<suffix> eg. config-3.5.x.y-xen0 in the above example. The scripts will automatically look for the Xen Dom0 options in kernel config and append Xen hypervisor boot lines to the grub menu. Note that for this to function correctly the kernel config file must be located in one of the following directories with a suffix matching the desired kernel:

Directory File Prefix Example
/etc/kernels/ kernel-config-* /etc/kernels/kernel-config-3.18.11-gentoo
/boot config-* /boot/config-3.18.11-gentoo

The example column above assumes a kernel named /boot/kernel-3.18.11-gentoo.

Alternatively, the following command should do the trick:

root #genkernel --oldconfig --menuconfig --install --bootloader=grub --symlink --disklabel --lvm --mdadm --makeopts=-j9 all

Now reboot your system into Xen and check if you can do whatever you normally do on your system. If this is the case, you can edit your bootloader configuration to always boot into Xen.

Note
If you wish to start guest domains automatically on boot add xendomains to the default runlevel as well and create a symlink in /etc/xen/auto/ to the Xen configuration files for the domains you wish to start.

Creating an Unpriviledged Domain (domU)

Building the Kernel

Go to the Xen-powered Linux kernel source and, if necessary, update the configuration. It is wise to keep as many topics as possible similar to the main kernel. Then build the kernel and place the resulting vmlinuz file where you want (we assume this is /mnt/data/xen/kernel ):

root #make O=~/build/domU
root #cp ~/build/domU/vmlinuz /mnt/data/xen/kernel/kernel-3.5.x.y-xen
Note
On modern systems (e.g. xen-4.3.2, kernel-3.12) it is possible to use only one kernel for both: dom0 and domU. Just configure all options needed for a guest system in the main kernel. Don't forget to copy the modules /lib/modules/<your kernel> to the guest system.

If you'd like to trim down the domU kernel the following flags are necessary.

Enable general Xen support:

KERNEL Xen Base
Processor type and features  --->
     [*] Linux guest support  --->
          [*]   Enable paravirtualization code
          [*]     Xen guest support
          [*]       Support for running as a PVH guest
          [*]     Paravirtualization layer for spinlocks

Facilitates guest access to block and network devices via dom0:

KERNEL Disk and Network
Device Drivers  --->
     [*] Block devices  --->
          <*>   Xen virtual block device support
     [*] Network device support  --->
          <*>   Xen network device frontend driver

In some configurations it can be desirable to provide a guest with direct access to a PCI device. This is known as Xen PCI Passthrough :

KERNEL Guest PCI Passthrough
Bus options (PCI etc.)  --->
     [*] Xen PCI Frontend

Keyboard, mouse, and display support via dom0 backend:

KERNEL Guest Human Interface
Device Drivers  --->
     Input device support  --->
          [*]   Miscellaneous devices  --->
               <*>   Xen virtual keyboard and mouse support
     Graphics support  --->
          Frame buffer Devices  --->
               <*> Xen virtual frame buffer support

The remaining drivers flesh out memory management, domain-to-domain communication, and communication to Xen via sysfs interfaces:

KERNEL Xen Drivers
Device Drivers  --->
     Xen driver support  --->
          [*] Xen memory balloon driver
          [*]   Scrub pages before returning them to system
          <*> Xen /dev/xen/evtchn device
          <*> Xen filesystem
          [*]   Create compatibility mount point /proc/xen
          [*] Create xen entries under /sys/hypervisor
          <*> userspace grant access device driver
          <*> User-space grant reference allocator driver
          <*> Xen ACPI processor
          [*] Xen platform mcelog

Creating the Domain Disks

For best performance, it is best to dedicate a partition (or logical volume) to a domain rather than a file based filesystem. However, if you are going to use Xen primarily for tests using a file based filesystem does have its advantages (especially regarding maintenance).

You can create a file based filesystem using dd and mke2fs (or any other file system creation tool). For instance, to create a 4 Gbyte ext4 filesystem:

root #dd if=/dev/zero of=/mnt/data/xen/disks/ext4root.img bs=1M count=4096
root #mkfs.ext4 /mnt/data/xen/disks/ext4root.img

Configuring a Domain

Next we create a Xen configuration file for a domain. You can store these configuration files where you want, for instance at /mnt/data/xen/configs . As an example, we create a configuration file for a small Gentoo environment which uses the disk image we created previously:

FILE /mnt/data/xen/configs/gentoo
kernel = "/mnt/data/xen/kernel/kernel-3.5.x.y-xen"
memory = 512
name   = "gentoo"
(Map the disk image to the virtual /dev/sda1)
disk   = ['file:/mnt/data/xen/disks/ext4root.img,sda1,w']
root   = "/dev/sda1 ro"

If you are using a block device (such as an lvm volume or partition) for the disk use 'phy:' instead of 'file:' and leave off /dev. For example:

CODE Using a block device
(LVM Volume)
disk = [ 'phy:lvm/xen-guest-root,sda1,w' ]
 
(Physical Partition)
disk = [ 'phy:sdb6,sda1,w' ]

You can find example configuration files in /etc/xen, which is also the default location for domU config files.

Launching the New Domain

Now we're all set and we can launch the new domain. If the disk image contained an operating system, we could just create and attach the domain using the xl command:

root #xl create /mnt/data/xen/gentoo -c

The domain would be booted inside the terminal in which you executed the command. However, in our case, the disk image is empty so the domain won't boot up in anything useful. To fix this, you can loop-mount the image and install Gentoo as you're used to.

If you want to disconnect from the domain, press Ctrl+] . You can always reconnect to the domains' console using xl console gentoo . However, there is only one console per domain, so only use it when you can't access the domain otherwise (for instance, through SSH).

If you are missing login prompt on the console, make sure you have entries like this in your inittab files on dom0 and domU pointing to /dev/hvc0:

FILE dom0:/etc/inittab
c0:2345:respawn:/sbin/agetty 38400 hvc0 linux
FILE domU:/etc/inittab
c0:2345:respawn:/sbin/agetty 38400 hvc0 linux

To apply the changes in /etc/inittab without a reboot issue the following command:

root #telinit q

If it still does not work, check the kernel config for the entries: DEVTMPFS and DEVTMPFS_MOUNT. They should be set.

KERNEL Kernel Config DEVTMPFS
Device Drivers --->
    Generic Driver Options  --->
        -*- Maintain a devtmpfs filesystem to mount at /dev
        [*]   Automount devtmpfs at /dev, after the kernel mounted the rootfs

Networking on Domains

Introduction

Xen works best when using a bridged mode network configuration. This means that your default network interface on the administrative domain becomes a bridge which accepts connections to the virtual domains as well as to the IP address your administrative domain has.

Bridged Interfaces

Create a bridge interface by creating a new link to the networking init script as provided by Gentoo:

root #cd /etc/init.d
root #ln -s net.lo net.br0

Next, edit /etc/conf.d/net and setup the bridge:

FILE dom0:/etc/conf.d/net
# eth0 should NOT have an ip configured
config_eth0="null"
 
# configure bridge to replace eth0 on dom0. Make sure the netmask for the bridge includes ip addresses of all your domU's!
bridge_br0="eth0"
config_br0="192.168.XX.XX netmask 255.255.0.0 brd 192.168.255.255"
routes_br0="default via 192.168.XX.XX"
mac_br0="00:16:3e:5b:XX:XX"
 
# bridge options to make interface coming up immediately
brctl_br0="stp off
        setfd 0
        sethello 10"
 
rc_net_br0_need="net.eth0"
rc_net_br0_provide="!net"
FILE domU:/etc/conf.d/net
config_eth0="192.168.1.200 netmask 255.255.255.0 brd 192.168.1.255"
# route all trafic through dom0 bridge address
routes_eth0="default via 192.168.XX.XX"
# make sure all your domU's have different mac addresses! Set them if needed in xen domU 
# config files in /etc/xen/<domU_name> with "vif = [ "ip=192.68.XX.XX,mac=XX:XX:XX:XX:XX:XX,bridge=br0" ];" !

Finally, install the net-misc/bridge-utils package, and make sure the net.br0 init script is loaded at boot.

root #emerge net-misc/bridge-utils
root #rc-update add net.br0 default

If you use bridged networks with real internet IP's in hosted environments, it may be necessary to add one or all of the following lines (depending on your environment) in your /etc/sysctl.conf file to prevent redirects, that can cause intermittent network interruptions:

FILE /etc/sysctl.conf
net.ipv4.conf.all.send_redirects=0
net.ipv4.conf.eth0.send_redirects=0
net.ipv4.conf.br0.send_redirects=0
net.ipv4.conf.default.send_redirects=0

To get the changes in /etc/sysctl.conf to work, use:

root #sysctl -p /etc/sysctl.conf

if you encounter a poor network performance or if your domU network permanently stops working under heavy load (backup jobs, etc) (from outside it looks, like the instance would crash, but deactivating and activating the interface e.g. form the xl console <domU name> with /etc/init.d/net.eth0 stop/start, restores normal operation) , use ethtool to improve/prevent it on all interfaces connected to the bridge (don't forget the bridge himself):

root #ethtool --offload <network device> gso off tso off sg off gro off
Note
You have to do it after each reboot, so use e.g. /etc/crontab to make it permanent.

Further Resources

Xen Documentation

Xen Tools

Xen Tuning

  • Xen network tuning
    This article is based on a document formerly found on our main website gentoo.org.
    The following people have contributed to the original document: Sven Vermeulen, nightmorph
    They are listed here as the Wiki history does not provide for any attribution. If you edit the Wiki article, please do not add yourself here, your contributions are recorded on the history page.