Embedded Handbook/General/Compiling with qemu user chroot

From Gentoo Wiki
Jump to:navigation Jump to:search
General topics
Introduction
Creating a cross-compiler
Cross-compiling with Portage
Cross-compiling the kernel
Compiling with qemu user chroot
Frequently asked questions
Emulators
Qemu
Armulator
Hercules
Bootloaders
Das U-Boot
NeTTrom
RedBoot
SH-LILO
Boards
Hammer Board and Nail Board
LANTank
NetWinder
NSLU2
QNAP TurboStation 109/209/409
Marvell Sheevaplug
ACME SYSTEMS Netus G20
Genesi Efika MX
Pandaboard
TrimSlice
BeagleBone
BeagleBone Black
Intel Edison


Cross compiling software with a QEMU user chroot.

Prerequsites

Kernel

The host system kernel will need support for miscellaneous binary formats. This is achived via the binfmt_misc module. Add the CONFIG_BINFMT_MISC=m or CONFIG_BINFMT_MISC=y symbols to the host's kernel .config file. If this module is not built already, then the development host will require a reboot after the kernel update and modules_install.

KERNEL Enable CONFIG_BINFMT_MISC
Executable file formats  --->
  <*> Kernel support for MISC binaries

Package configuration

In order to take advantage of QEMU user mode a few steps are necessary. First the app-emulation/qemu package must be emerged with the right settings. Specifically this means building with USE=static-user and setting QEMU_SOFTMMU_TARGETS and QEMU_USER_TARGETS to include the targets that will be utilized.

See man portage(5) for other ways of doing this:

root #echo "app-emulation/qemu static-user" >> /etc/portage/package.use/qemu

Activating the static-user will require supporting libraries to be build with static-libs support:

FILE /etc/portage/package.use/static-libsConfiguring qemu supporting libraries with static-libs
# required by app-emulation/qemu::gentoo[static,static-user]
# required by qemu (argument)
dev-libs/glib static-libs
# required by app-emulation/qemu::gentoo[-static,static-user]
# required by qemu (argument)
sys-libs/zlib static-libs
# required by app-emulation/qemu::gentoo[-static,static-user,xattr]
# required by qemu (argument)
sys-apps/attr static-libs
# required by dev-libs/glib::gentoo
# required by app-emulation/qemu::gentoo[-static,static-user]
# required by qemu (argument)
dev-libs/libpcre2 static-libsx
Note
GCC will use huge amount of RAM when LTO is enabled on the system while using the static-user flag, because of this is recommended to disable LTO while compiling in this configuration or use clang if LTO is required. See bug #883419

Tweak the list here to include the necessary target(s). See the output of emerge -pv app-emulation/qemu for the full list:

Important
For Raspberry Pi running a 32-bit stage3 build, be sure to select QEMU_USER_TARGETS="arm" here. For 64-bit stage3 builds, select aarch64. When in doubt set both.

To build all targets:

root #echo 'QEMU_SOFTMMU_TARGETS="alpha aarch64 arm i386 loongarch64 m68k mips mips64 mips64el mipsel ppc ppc64 riscv32 riscv64 s390x sh4 sh4eb sparc sparc64 x86_64"' >> /etc/portage/make.conf
root #echo 'QEMU_USER_TARGETS="alpha aarch64 arm armeb i386 loongarch64 m68k mips mipsel ppc ppc64 ppc64abi32 riscv32 riscv64 s390x sh4 sh4eb sparc sparc32plus sparc64"' >> /etc/portage/make.conf

Alternatively, the USE_EXPAND syntax can be used to keep make.conf free of USE_EXPAND values:

FILE /etc/portage/package.use/qemuConfigure qemu for user chroot
app-emulation/qemu static-user QEMU_SOFTMMU_TARGETS: alpha aarch64 arm i386 loongarch64 m68k mips mips64 mips64el mipsel ppc ppc64 riscv32 riscv64 s390x sh4 sh4eb sparc sparc64 x86_64
app-emulation/qemu QEMU_USER_TARGETS: alpha aarch64 arm armeb i386 loongarch64 m68k mips mipsel ppc ppc64 ppc64abi32 riscv32 riscv64 s390x sh4 sh4eb sparc sparc32plus sparc64 x86_64

Then install the package:

root #emerge --ask --update --newuse --deep app-emulation/qemu

At this point, it is wise to create a binary package for QEMU:

root #quickpkg app-emulation/qemu

Finally, for non-root users to use qemu, they will need added to the kvm group. Replace larry in the example below with the appropriate username:

root #gpasswd -a larry kvm

Register binary format handlers

Mount the binfmt_misc handler if it is not already mounted, then register the format with the kernel via the procfs:

root #[ -d /proc/sys/fs/binfmt_misc ] || modprobe binfmt_misc
root #[ -f /proc/sys/fs/binfmt_misc/register ] || mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc

Do not register a handler that matches the host machine (in this example, registers for x86 and AMD64 has been excluded since the host machine).

32-bit registers:

root #echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm:' > /proc/sys/fs/binfmt_misc/register
root #echo ':armeb:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-armeb:' > /proc/sys/fs/binfmt_misc/register
root #echo ':alpha:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x26\x90:\xff\xff\xff\xff\xff\xfe\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-alpha:' > /proc/sys/fs/binfmt_misc/register
root #echo ':mips:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-mips:' > /proc/sys/fs/binfmt_misc/register
root #echo ':mipsel:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-mipsel:' > /proc/sys/fs/binfmt_misc/register
root #echo ':ppc:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x14:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-ppc:' > /proc/sys/fs/binfmt_misc/register
root #echo ':sh4:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x2a\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfb\xff\xff\xff:/usr/bin/qemu-sh4:' >/proc/sys/fs/binfmt_misc/register
root #echo ':sh4eb:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x2a:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-sh4eb:' >/proc/sys/fs/binfmt_misc/register
root #echo ':sparc:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x02:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-sparc:' > /proc/sys/fs/binfmt_misc/register
root #echo ':m68k:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x04:\xff\xff\xff\xff\xff\xff\xfe\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-m68k:' > /proc/sys/fs/binfmt_misc/register

MIPS n32 registers:

root #echo ':mipsn32:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-mipsn32:' > /proc/sys/fs/binfmt_misc/register
root #echo ':mipseln32:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-mipseln32:' > /proc/sys/fs/binfmt_misc/register

Add 64-bit registers:

root #echo ':aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\xfc\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64:' > /proc/sys/fs/binfmt_misc/register
root #echo ':loongarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x02\x01:\xff\xff\xff\xff\xff\xff\xff\xfc\x00\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:qemu-system-loongarch64:' > /proc/sys/fs/binfmt_misc/register
root #echo ':ppc64:M::\x7fELF\x02\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x15:\xff\xff\xff\xff\xff\xff\xff\xfc\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-ppc64:' > /proc/sys/fs/binfmt_misc/register
root #echo ':riscv64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xf3\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-riscv64:' > /proc/sys/fs/binfmt_misc/register
root #echo ':mips64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xf3\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-mips64:' > /proc/sys/fs/binfmt_misc/register

Services

After this, make sure the binfmt service is (re-)started:

OpenRC

root #rc-service qemu-binfmt start

It may be wise for the services to be started by default on boot:

root #rc-update add qemu-binfmt default

systemd

For the systemd-binfmt service, add files containing the desired handler registration strings under /etc/binfmt.d/. Modern versions of qemu ship a binfmt configuration file that supports all binary formats. Simply link it to /etc for binary format support, then skip the following manual file creation steps.

root #ln -s /usr/share/qemu/binfmt.d/qemu.conf /etc/binfmt.d/qemu.conf

Alternatively, to only make support for aarch64 and arm architectures, files separate files can manually be created:

FILE /etc/binfmt.d/qemu-aarch64-static.confSpecifying an aarch64 binfmt-misc handler for systemd-binfmt
:aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\xfc\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64:
FILE /etc/binfmt.d/qemu-arm-static.confSpecifying an arm binfmt-misc handler for systemd-binfmt
:arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm:
Note
In the FileBox above, the eighth byte of the mask has been set to fc (rather than ff), to allow programs like gcc to be executed. See for example Debian bug #799120.

With the file(s) created in /etc/binfmt.d/, systemd will now find them automatically at system boot time. See man 5 binfmt.d for more information.

Since the files were likely created just a few moments ago, it will be necessary to restart the service once to register the binary formats:

root #systemctl restart systemd-binfmt

To confirm the service is running properly after restarting:

root #systemctl status systemd-binfmt

Setup chroot

Download the desired stage tarball:

root #wget http://arch-stageball

Unpack the tarball:

root #tar -xzvf arch-stageball
root #cd arch-stageball

Install the static QEMU into the chroot:

root #ROOT=$PWD/ emerge --usepkgonly --oneshot --nodeps qemu
root #mkdir -p var/db/repos/gentoo

Mount the required directories:

root #mount --bind /var/db/repos/gentoo var/db/repos/gentoo
root #mount --bind /proc proc
root #mount --bind /sys sys
root #mount --bind /dev dev
root #mount --bind /dev/pts dev/pts
root #mount --bind /dev/shm dev/shm

Chroot into the environment:

root #chroot . /bin/bash --login

Keep QEMU from being altered within the chroot (if deleted or reinstalled within the chroot, breakage will occur):

root #echo 'EMERGE_DEFAULT_OPTS="$EMERGE_DEFAULT_OPTS --exclude app-emulation/qemu"' >> /etc/portage/make.conf
root #echo app-emulation/qemu >> /var/lib/portage/world

Unmount the various file systems when they are no longer in use:

root #umount var/db/repos/gentoo sys proc dev/pts dev

Sometimes we'll need to pass additional args to QEMU (CPU model), so we'll create a wrapper script (in C) that'll call QEMU with it:

FILE qemu-wrapper.c
/*
 * Pass arguments to qemu binary
 */

#include <string.h>
#include <unistd.h>

int main(int argc, char **argv, char **envp) {
	char *newargv[argc + 3];

	newargv[0] = argv[0];
	newargv[1] = "-cpu";
	newargv[2] = "cortex-a8"; /* here you can set the cpu you are building for */

	memcpy(&newargv[3], &argv[1], sizeof(*argv) * (argc -1));
	newargv[argc + 2] = NULL;
	return execve("/usr/bin/qemu-arm", newargv, envp);
}

Compile the wrapper with:

root #gcc -static qemu-wrapper.c -O3 -s -o qemu-wrapper

Then copy into the chroot. Notice the first example ARM entry in the binfmt_misc section uses this method.

Caveat

Currently QEMU does not support the pid-sandbox (bug #703278) and network-sandbox (bug #703276) features of Portage.

This issue can be worked around by setting the FEATURES environment variable before each call to emerge and other Portage tools in the chroot:

root #env FEATURES="-pid-sandbox -network-sandbox" emerge
root #env FEATURES="-pid-sandbox -network-sandbox" equery

Alternatively, modify Portage's FEATURES variable within the chroot.

FILE /etc/portage/make.conf
FEATURES="-pid-sandbox -network-sandbox"


← Cross-compiling the kernel Home Frequently asked questions →

This page is based on a document formerly found on our main website gentoo.org.
The following people contributed to the original document: Mike Frysinger, Ned Ludd, Robin H. Johnson, Alex Tarkovsky, Alexey Shvetsov, Raúl Porcel, Joshua Saddler on April 28, 2013.
They are listed here because wiki history does not allow for any external attribution. If you edit the wiki article, please do not add yourself here; your contributions are recorded on each article's associated history page.

Retrieved from "https://wiki.gentoo.org/index.php?title=Embedded_Handbook/General/Compiling_with_qemu_user_chroot&oldid=1260283"