Embedded Handbook/General/Compiling with QEMU user chroot

From Gentoo Wiki
Jump to:navigation Jump to:search
General topics
Introduction
Compiling with QEMU user chroot
Creating a cross-compiler
Cross-compiling with Portage
Cross-compiling the kernel
Frequently asked questions
Emulators
Qemu
Armulator
Hercules
Bootloaders
Das U-Boot
NeTTrom
RedBoot
SH-LILO
Boards
Hammer Board and Nail Board
LANTank
NetWinder
NSLU2
QNAP TurboStation 109/209/409
Marvell Sheevaplug
ACME SYSTEMS Netus G20
Genesi Efika MX
Pandaboard
TrimSlice
BeagleBone
BeagleBone Black
Intel Edison


Compiling with QEMU user chroot.

This wiki page explains the required steps needed to be able to chroot into a root filesystem whose platform (e.g. aarch64) is different from the running system (e.g. amd64). Software can then be compiled within the chroot transparently using QEMU emulation.

Installation

Kernel

The build system's kernel must support miscellaneous binary formats. This can be enabled with CONFIG_BINFMT_MISC=m or CONFIG_BINFMT_MISC=y in the the kernel's .config file.

Tip
A system restart is required after building this module before it can be used.
KERNEL Enable CONFIG_BINFMT_MISC
Executable file formats  --->
  <*> Kernel support for MISC binaries

USE Flags

USE flags for app-emulation/qemu QEMU + Kernel-based Virtual Machine userland tools

accessibility Adds support for braille displays using brltty
aio Enables support for Linux's Async IO
alsa Enable alsa output for sound emulation
bpf Enable eBPF support for RSS implementation.
bzip2 Enable bzip2 compression support
capstone Enable disassembly support with dev-libs/capstone
curl Support ISOs / -cdrom directives via HTTP or HTTPS.
debug Enable extra debug codepaths, like asserts and extra output. If you want to get meaningful backtraces see https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Backtraces
doc Add extra documentation (API, Javadoc, etc). It is recommended to enable per package instead of globally
fdt Enables firmware device tree support
filecaps Use Linux file capabilities to control privilege rather than set*id (this is orthogonal to USE=caps which uses capabilities at runtime e.g. libcap)
fuse Enables FUSE block device export
glusterfs Enables GlusterFS cluster fileystem via sys-cluster/glusterfs
gnutls Enable TLS support for the VNC console server. For 1.4 and newer this also enables WebSocket support. For 2.0 through 2.3 also enables disk quorum support.
gtk Add support for x11-libs/gtk+ (The GIMP Toolkit)
infiniband Enable Infiniband RDMA transport support
io-uring Enable efficient I/O via sys-libs/liburing.
iscsi Enable direct iSCSI support via net-libs/libiscsi instead of indirectly via the Linux block layer that sys-block/open-iscsi does.
jack Add support for the JACK Audio Connection Kit
jemalloc Enable jemalloc allocator support
jpeg Enable jpeg image support for the VNC console server
keyutils Support Linux keyrings via sys-apps/keyutils
lzo Enable support for lzo compression
multipath Enable multipath persistent reservation passthrough via sys-fs/multipath-tools.
ncurses Enable the ncurses-based console
nfs Enable NFS support
nls Add Native Language Support (using gettext - GNU locale utilities)
numa Enable NUMA support
opengl Add support for OpenGL (3D graphics)
oss Add support for OSS (Open Sound System)
pam Add support for PAM (Pluggable Authentication Modules) - DANGEROUS to arbitrarily flip
pin-upstream-blobs Pin the versions of BIOS firmware to the version included in the upstream release. This is needed to sanely support migration/suspend/resume/snapshotting/etc... of instances. When the blobs are different, random corruption/bugs/crashes/etc... may be observed.
pipewire Enable pipewire output for sound emulation
plugins Enable qemu plugin API via shared library loading.
png Enable png image support for the VNC console server
pulseaudio Enable pulseaudio output for sound emulation
python Add optional support/bindings for the Python language
rbd Enable rados block device backend support, see https://docs.ceph.com/en/mimic/rbd/qemu-rbd/
sasl Add support for the Simple Authentication and Security Layer
sdl Enable the SDL-based console
sdl-image SDL Image support for icons
seccomp Enable seccomp (secure computing mode) to perform system call filtering at runtime to increase security of programs
selinux !!internal use only!! Security Enhanced Linux support, this must be set by the selinux profile or breakage will occur
slirp Enable TCP/IP in hypervisor via net-libs/libslirp
smartcard Enable smartcard support
snappy Enable support for Snappy compression (as implemented in app-arch/snappy)
spice Enable Spice protocol support via app-emulation/spice
ssh Enable SSH based block device support via net-libs/libssh2
static Build the User and Software MMU (system) targets as well as tools as static binaries
static-user Build the User targets as static binaries
systemtap Enable SystemTAP/DTrace tracing
test Enable dependencies and/or preparations necessary to run tests (usually controlled by FEATURES=test but can be toggled independently)
udev Enable virtual/udev integration (device discovery, power and storage device support, etc)
usb Enable USB passthrough via dev-libs/libusb
usbredir Use sys-apps/usbredir to redirect USB devices to another machine over TCP
vde Enable VDE-based networking
vhost-net Enable accelerated networking using vhost-net, see https://www.linux-kvm.org/page/VhostNet
virgl Enable experimental Virgil 3d (virtual software GPU)
virtfs Enable VirtFS via virtio-9p-pci / fsdev. See https://wiki.qemu.org/Documentation/9psetup
vnc Enable VNC (remote desktop viewer) support
vte Enable terminal support (x11-libs/vte) in the GTK+ interface
xattr Add support for getting and setting POSIX extended attributes, through sys-apps/attr. Requisite for the virtfs backend.
xen Enables support for Xen backends
zstd Enable support for ZSTD compression

Important
app-emulation/qemu must be emerged with USE=static-user to use user targets.
Warning
QEMU_SOFTMMU_TARGETS and QEMU_USER_TARGETS are empty by default and must be defined to utilize user targets.
Note
Activating the static-user will require supporting libraries to be build with static-libs support.
FILE /etc/portage/package.use/qemuEnable aarch64 user target and static-libs in supporting libraries.
# Enable static-user and add the aarch64 target
app-emulation/qemu static-user QEMU_SOFTMMU_TARGETS: aarch64 QEMU_USER_TARGETS: aarch64
# required by app-emulation/qemu::gentoo[static,static-user]
# required by qemu (argument)
dev-libs/glib static-libs
# required by app-emulation/qemu::gentoo[-static,static-user]
# required by qemu (argument)
sys-libs/zlib static-libs
# required by app-emulation/qemu::gentoo[-static,static-user,xattr]
# required by qemu (argument)
sys-apps/attr static-libs
# required by dev-libs/glib::gentoo
# required by app-emulation/qemu::gentoo[-static,static-user]
# required by qemu (argument)
dev-libs/libpcre2 static-libs
Warning
If using LTO and the static-user, GCC will use huge amount of RAM when compiling. Because of this, it is generally recommended to disable LTO while compiling in this configuration or use Clang if LTO is required.
See also
bug #883419

QEMU target configuration

By default, app-emulation/qemu does not define any QEMU_SOFTMMU_TARGETS or QEMU_USER_TARGETS. The example configuration above only includes aarch64 targets.

Tip
The output of emerge -pv app-emulation/qemu contains the full target list.
Tip
To build for a Raspberry Pi running a 32-bit stage3 build, be sure to select QEMU_USER_TARGETS="arm" here. For 64-bit stage3 builds, select aarch64. When in doubt both can be set.

To build all targets:

root #echo 'QEMU_SOFTMMU_TARGETS="*"' >> /etc/portage/make.conf
root #echo 'QEMU_USER_TARGETS="*"' >> /etc/portage/make.conf

Alternatively, the USE_EXPAND syntax can be used to keep make.conf free of USE_EXPAND values:

FILE /etc/portage/package.use/qemuConfigure QEMU to build all targets.
app-emulation/qemu QEMU_SOFTMMU_TARGETS: * QEMU_USER_TARGETS: *

Emerge

root #emerge --ask --update --newuse --deep app-emulation/qemu

Configuration

Group configuration

For non-root users to use QEMU, they must be added to the kvm group.

To add larry to the kvm group:

root #gpasswd -a larry kvm

OpenRC

To start the qemu-binfmt service:

root #rc-service qemu-binfmt start

It may be wise for the services to be started by default on boot:

root #rc-update add qemu-binfmt default

systemd

The systemd-binfmt service must be configured by adding files containing the desired handler registration strings to /etc/binfmt.d/.

Modern versions of qemu ship a binfmt configuration file that supports all binary formats. Simply link it to /etc for binary format support, then skip the following manual file creation steps.

root #ln -s /usr/share/qemu/binfmt.d/qemu.conf /etc/binfmt.d/qemu.conf

Manual binfmt configuration

Alternatively, to only make support for aarch64 and arm architectures, separate files can manually be created:

FILE /etc/binfmt.d/qemu-aarch64-static.confSpecifying an aarch64 binfmt-misc handler for systemd-binfmt
:aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\xfc\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64:
FILE /etc/binfmt.d/qemu-arm-static.confSpecifying an arm binfmt-misc handler for systemd-binfmt
:arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm:

See #Binary format handlers for a list of possible handlers.

Note
In the FileBox above, the eighth byte of the mask has been set to fc (rather than ff), to allow programs like gcc to be executed. See for example Debian bug #799120.

With the file(s) created in /etc/binfmt.d/, systemd will now find them automatically at system boot time. See man 5 binfmt.d for more information.

Since the files were likely created just a few moments ago, it will be necessary to restart the service once to register the binary formats:

root #systemctl restart systemd-binfmt

To confirm the service is running properly after restarting:

root #systemctl status systemd-binfmt

Usage

Preparing the chroot

To be able to chroot into a system of a different platform (e.g. aarch64 while using an amd64 system), mounted in e.g. /mnt/gentoo, the QEMU static-user binary must be copied into the environment.

This file is named qemu-<architecture> under /usr/bin/ and should be copied to usr/bin/ within the build environment.

To use an aarch64 chroot environment at /mnt/gentoo-aarch64:

user $cp /usr/bin/qemu-aarch64 /mnt/gentoo-aarch64/usr/bin

Chrooting

Once the environment has been prepared, sys-apps/arch-chroot can be used:

root #arch-chroot /mnt/gentoo-aarch64

Portage configuration

Currently QEMU does not support the pid-sandbox (bug #703278) and network-sandbox (bug #703276) Portage features.

To disable these features::

FILE /etc/portage/make.conf
FEATURES="-pid-sandbox -network-sandbox"

Additional Usage

Binary format handlers

Binary format handlers look rather confusing at first, but they can be simply broken down as: :name:type:offset:magic:mask:interpreter:flags[1]

With each field representing, in order of appearance:

  • name being the name of the binary format, a new /proc file will be created with this name at /proc/sys/fs/binfmt_misc
  • type being either E or M
    • E means the executable file format is identified by its file extension
    • M means the format is identified by the magic number mask
  • offset represents the offset of the magic/mask in the file in bytes, the default is 0
  • magic is a byte sequence that binfmt matches for to know what files to pass through to the interpreter
  • interpreter is a program that is to be run with the matching file as an argument
  • flags is a field that controls different aspects of interpreter when it is invoked
    • P - preserve-argv[0]: adds an argument to the argument vector to preserve the original argv[0]
    • O - open-binary: opens the file for reading and pass its descriptor as an argument rather than the full path
    • C - credentials: current behavior of binfmt_misc is to calculate the credentials and security of the new process according to the interpreter, but including this flag will be calculated according to the binary. (Implies the O flag)
    • F - fix binary: currently, the binary is spawned lazily when the misc format file is invoked, however, this does not work very well in the face of mount namespaces and changroots so F allows the binary to be opened always once it is installed, regardless of environment changes.

Some restrictions do apply with binfmt:

  • The whole register string may not exceed 1920 characters
  • The magic must reside in the first 128 bytes of the file
  • The interpreter string may not exceed 127 characters
Architecture
Binfmt Handler String
AARCH64 :aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\xfc\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64:
ARM :arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm:
ARMeb :armeb:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-armeb:
Alpha :alpha:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x26\x90:\xff\xff\xff\xff\xff\xfe\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-alpha:
DOS :DOSWin:M::MZ::/etc/eselect/wine/bin/wine:
LOONGARCH64 :loongarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x02\x01:\xff\xff\xff\xff\xff\xff\xff\xfc\x00\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-loongarch64:
M68K :m68k:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x04:\xff\xff\xff\xff\xff\xff\xfe\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-m68k:
MIPS :mips:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-mips:
MIPS64 :mips64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xf3\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-mips64:
MIPSel :mipsel:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-mipsel:
MIPS N32 :mipsn32:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-mipsn32:
MIPSel N32 :mipseln32:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-mipseln32:
PPC :ppc:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x14:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-ppc:
PPC64 :ppc64:M::\x7fELF\x02\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x15:\xff\xff\xff\xff\xff\xff\xff\xfc\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-ppc64:
RISCV64 :riscv64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xf3\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-riscv64:
SH4 :sh4:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x2a\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfb\xff\xff\xff:/usr/bin/qemu-sh4:
SH4eb :sh4eb:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x2a:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-sh4eb:
SPARC :sparc:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x02:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-sparc:

Manual setup

First, mount the binfmt_misc handler if it is not already mounted, then register the format with the kernel via the procfs:

root #[ -d /proc/sys/fs/binfmt_misc ] || modprobe binfmt_misc
root #[ -f /proc/sys/fs/binfmt_misc/register ] || mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc
Important
Do not register a handler that matches the host machine (in this example, registers for x86 and AMD64 has been excluded since the host machine).

To register a handler, pipe its format to /proc/sys/fs/binfmt_misc/register, for example

root #echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm:' > /proc/sys/fs/binfmt_misc/register
See also
#Binary format handlers for a list of possible handlers.

Advanced Tips

Sometimes we'll need to pass additional args to QEMU (CPU model), so we'll create a wrapper script (in C) that'll call QEMU with it:

FILE qemu-wrapper.c
/*
 * Pass arguments to qemu binary
 */

#include <string.h>
#include <unistd.h>

int main(int argc, char **argv, char **envp) {
	char *newargv[argc + 3];

	newargv[0] = argv[0];
	newargv[1] = "-cpu";
	newargv[2] = "cortex-a8"; /* here you can set the cpu you are building for */

	memcpy(&newargv[3], &argv[1], sizeof(*argv) * (argc -1));
	newargv[argc + 2] = NULL;
	return execve("/usr/bin/qemu-arm", newargv, envp);
}

Compile the wrapper with:

root #gcc -static qemu-wrapper.c -O3 -s -o qemu-wrapper

Then copy into the chroot. Notice the first example ARM entry in the binfmt_misc section uses this method.



This page is based on a document formerly found on our main website gentoo.org.
The following people contributed to the original document: Mike Frysinger, Ned Ludd, Robin H. Johnson, Alex Tarkovsky, Alexey Shvetsov, Raúl Porcel, Joshua Saddler on April 28, 2013.
They are listed here because wiki history does not allow for any external attribution. If you edit the wiki article, please do not add yourself here; your contributions are recorded on each article's associated history page.