Embedded Handbook/General/Compiling with qemu user chroot

From Gentoo Wiki
Jump to:navigation Jump to:search
General topics
Introduction
Creating a cross-compiler
Cross-compiling with Portage
Cross-compiling the kernel
Compiling with qemu user chroot
Frequently asked questions
Emulators
Qemu
Armulator
Hercules
Bootloaders
Das U-Boot
NeTTrom
RedBoot
SH-LILO
Boards
Hammer Board and Nail Board
LANTank
NetWinder
NSLU2
QNAP TurboStation 109/209/409
Marvell Sheevaplug
ACME SYSTEMS Netus G20
Genesi Efika MX
Pandaboard
TrimSlice
BeagleBone
BeagleBone Black
Intel Edison


Cross compiling software with a QEMU user chroot.

Prerequsites

Kernel

The host system kernel will need support for miscellaneous binary formats. This is achived via the binfmt_misc module. Add the CONFIG_BINFMT_MISC=m or CONFIG_BINFMT_MISC=y symbols to the host's kernel .config file. If this module is not built already, then the development host will require a reboot after the kernel update and modules_install.

KERNEL Enable CONFIG_BINFMT_MISC
Executable file formats  --->
  <*> Kernel support for MISC binaries

Package configuration

In order to take advantage of QEMU user mode a few steps are necessary. First the app-emulation/qemu package must be emerged with the right settings. Specifically this means building with USE=static-user and setting QEMU_USER_TARGETS to include the targets that will be utilized.

See man portage(5) for other ways of doing this:

root #echo app-emulation/qemu static-user >> /etc/portage/package.use

Tweak the list here to include the necessary target(s). See the output of emerge -pv app-emulation/qemu for the full list:

Important
For Raspberry Pi running a 32-bit stage3 build, be sure to select QEMU_USER_TARGETS="arm" here. For 64-bit stage3 builds, select aarch64. When in doubt set both.

To build all targets:

root #echo 'QEMU_SOFTMMU_TARGETS="alpha aarch64 arm i386 m68k mips mips64 mips64el mipsel ppc ppc64 s390x sh4 sh4eb sparc sparc64 x86_64"' >> /etc/portage/make.conf
root #echo 'QEMU_USER_TARGETS="alpha aarch64 arm armeb i386 m68k mips mipsel ppc ppc64 ppc64abi32 s390x sh4 sh4eb sparc sparc32plus sparc64"' >> /etc/portage/make.conf

Then install the package:

root #emerge --ask app-emulation/qemu

At this point it is wise to create a binary package for QEMU:

root #quickpkg app-emulation/qemu

Register binary format handlers

Mount the binfmt_misc handler if it is not already mounted, then register the format with the kernel via the procfs:

root #[ -d /proc/sys/fs/binfmt_misc ] || modprobe binfmt_misc
root #[ -f /proc/sys/fs/binfmt_misc/register ] || mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc

Do not register a handler that matches the host machine (in this example, registers for x86 and AMD64 has been excluded since the host machine).

32-bit registers:

root #echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm:' > /proc/sys/fs/binfmt_misc/register
root #echo ':armeb:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-armeb:' > /proc/sys/fs/binfmt_misc/register
root #echo ':alpha:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x26\x90:\xff\xff\xff\xff\xff\xfe\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-alpha:' > /proc/sys/fs/binfmt_misc/register
root #echo ':mips:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-mips:' > /proc/sys/fs/binfmt_misc/register
root #echo ':mipsel:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-mipsel:' > /proc/sys/fs/binfmt_misc/register
root #echo ':ppc:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x14:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-ppc:' > /proc/sys/fs/binfmt_misc/register
root #echo ':sh4:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x2a\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfb\xff\xff\xff:/usr/bin/qemu-sh4:' >/proc/sys/fs/binfmt_misc/register
root #echo ':sh4eb:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x2a:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-sh4eb:' >/proc/sys/fs/binfmt_misc/register
root #echo ':sparc:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x02:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-sparc:' > /proc/sys/fs/binfmt_misc/register
root #echo ':m68k:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x04:\xff\xff\xff\xff\xff\xff\xfe\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-m68k:' > /proc/sys/fs/binfmt_misc/register

Add 64-bit registers:

root #echo ':aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\xfc\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64:' > /proc/sys/fs/binfmt_misc/register
root #echo ':ppc64:M::\x7fELF\x02\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x15:\xff\xff\xff\xff\xff\xff\xff\xfc\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-ppc64:' > /proc/sys/fs/binfmt_misc/register
root #echo ':riscv64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xf3\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-riscv64:' > /proc/sys/fs/binfmt_misc/register

Services

After this, make sure the binfmt service is (re-)started:

OpenRC

root #rc-service qemu-binfmt start

It may be wise for the services to be started by default on boot:

root #rc-update add qemu-binfmt default

systemd

For the systemd-binfmt service, add files containing the desired handler registration strings under /etc/binfmt.d/. For example, for aarch64 and arm systems:

FILE /etc/binfmt.d/qemu-aarch64-static.confSpecifying an aarch64 binfmt-misc handler for systemd-binfmt
:aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\xfc\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64:
FILE /etc/binfmt.d/qemu-arm-static.confSpecifying an arm binfmt-misc handler for systemd-binfmt
:arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm:
Note
In the FileBox above, the eighth byte of the mask has been set to fc (rather than ff), to allow programs like gcc to be executed. See for example Debian bug #799120.

With the files created in /etc/binfmt.d/, systemd will now find them automatically at system boot time. See man 5 binfmt.d for more information.

Since the files were likely created just a few moments ago, it will be necessary to restart the service once to register the binary formats:

root #systemctl restart systemd-binfmt

To confirm the service is running properly after restarting:

root #systemctl status systemd-binfmt

Setup chroot

Download the desired stage tarball:

root #wget http://arch-stageball

Unpack the tarball:

root #tar -xzvf arch-stageball
root #cd arch-stageball

Install the static qemu into the chroot:

root #ROOT=$PWD/ emerge --usepkgonly --oneshot --nodeps qemu
root #mkdir -p usr/portage

Mount the required directories:

root #mount --bind /usr/portage usr/portage
root #mount --bind /proc proc
root #mount --bind /sys sys
root #mount --bind /dev dev
root #mount --bind /dev/pts dev/pts

Chroot into the environment:

root #chroot . /bin/bash --login

Keep QEMU from being altered within the chroot (if deleted or reinstalled within the chroot, breakage will occur):

root #echo 'EMERGE_DEFAULT_OPTS="$EMERGE_DEFAULT_OPTS --exclude app-emulation/qemu"' >>/etc/portage/make.conf
root #echo app-emulation/qemu >>/var/lib/portage/world

Unmount the various file systems when they are no longer in use:

root #umount usr/portage sys proc dev/pts dev

Sometimes we'll need to pass additional args to QEMU (CPU model), so we'll create a wrapper script (in C) that'll call QEMU with it:

FILE qemu-wrapper.c
/*
 * pass arguments to qemu binary
 */

#include <string.h>
#include <unistd.h>

int main(int argc, char **argv, char **envp) {
	char *newargv[argc + 3];

	newargv[0] = argv[0];
	newargv[1] = "-cpu";
	newargv[2] = "cortex-a8"; /* here you can set the cpu you are building for */

	memcpy(&newargv[3], &argv[1], sizeof(*argv) * (argc -1));
	newargv[argc + 2] = NULL;
	return execve("/usr/bin/qemu-arm", newargv, envp);
}

Compile the wrapper with:

root #gcc -static qemu-wrapper.c -O3 -s -o qemu-wrapper

Then copy into the chroot. Notice the first example ARM entry in the binfmt_misc section uses this method.

Caveat

Currently qemu does not support pid-sandbox and network-sandbox.

This issue can be worked around by setting the FEATURES environment variable before each call to emerge and other Portage tools in the chroot:

root #env FEATURES="-pid-sandbox -network-sandbox" emerge
root #env FEATURES="-pid-sandbox -network-sandbox" equery

Alternatively, modify Portage's FEATURES variable within the chroot.

FILE /etc/portage/make.conf
FEATURES="-pid-sandbox -network-sandbox"



This page is based on a document formerly found on our main website gentoo.org.
The following people contributed to the original document: Mike Frysinger, Ned Ludd, Robin H. Johnson, Alex Tarkovsky, Alexey Shvetsov, Raúl Porcel, Joshua Saddler on April 28, 2013.
They are listed here because wiki history does not allow for any external attribution. If you edit the wiki article, please do not add yourself here; your contributions are recorded on each article's associated history page.