Embedded Handbook/General/Compiling with qemu user chroot

From Gentoo Wiki
Jump to: navigation, search
General topics
Creating a cross-compiler
Cross-compiling with Portage
Cross-compiling the kernel
Compiling with qemu user chroot
Frequently asked questions
Das U-Boot
Hammer Board and Nail Board
QNAP TurboStation 109/209/409
Marvell Sheevaplug
Genesi Efika MX
BeagleBone Black
Intel Edison
External resources

How to compile with QEMU user chroot.


In order to take advantage of QEMU user mode we need to do a few things. First we need to emerge the app-emulation/qemu package with the right settings. That means building it with USE=static-user and setting QEMU_USER_TARGETS to include the targets we want to utilize.

See man portage(5) for other ways of doing this:

root #echo app-emulation/qemu static-user >> /etc/portage/package.use

Tweak the list here to include the target(s) you care about. See the output of emerge -pv qemu for the full list:

For Raspberry Pi builds, if you're doing a 32-bit stage3 build, you'll want to select arm here. For 64-bit stage3 builds, select aarch64.
root #echo 'QEMU_SOFTMMU_TARGETS="alpha aarch64 arm i386 mips mips64 mips64el mipsel ppc ppc64 s390x sh4 sh4eb sparc sparc64 x86_64"' >> /etc/portage/make.conf
root #echo 'QEMU_USER_TARGETS="alpha aarch64 arm armeb i386 mips mipsel ppc ppc64 ppc64abi32 s390x sh4 sh4eb sparc sparc32plus sparc64"' >> /etc/portage/make.conf

Then install the package:

root #emerge --ask app-emulation/qemu

and create a binary package for it

root #quickpkg app-emulation/qemu

Next we need to build the kernel module binfmt_misc. Add this to your kernel .config file: CONFIG_BINFMT_MISC=m or CONFIG_BINFMT_MISC=y. If this module is not built already, then the devel host will require a reboot after the kernel update and modules_install.

Mount the binfmt_misc handler if it's not already, then we need to register our format with the kernel via the procfs:

root #[ -d /proc/sys/fs/binfmt_misc ] || modprobe binfmt_misc
root #[ -f /proc/sys/fs/binfmt_misc/register ] || mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc

Do not register a handler that matches the host machine:

root #echo ':aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-aarch64:' > /proc/sys/fs/binfmt_misc/register
root #echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/qemu-wrapper:' > /proc/sys/fs/binfmt_misc/register
root #echo ':armeb:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-armeb:' > /proc/sys/fs/binfmt_misc/register
root #echo ':alpha:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x26\x90:\xff\xff\xff\xff\xff\xfe\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-alpha:' > /proc/sys/fs/binfmt_misc/register
root #echo ':mips:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-mips:' > /proc/sys/fs/binfmt_misc/register
root #echo ':mipsel:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-mipsel:' > /proc/sys/fs/binfmt_misc/register
root #echo ':ppc:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x14:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-ppc:' > /proc/sys/fs/binfmt_misc/register
root #echo ':sh4:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x2a\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfb\xff\xff\xff:/usr/bin/qemu-sh4:' >/proc/sys/fs/binfmt_misc/register
root #echo ':sh4eb:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x2a:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-sh4eb:' >/proc/sys/fs/binfmt_misc/register
root #echo ':sparc:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x02:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-sparc:' > /proc/sys/fs/binfmt_misc/register

After this, make sure the binfmt service is (re-)started:


root #/etc/init.d/qemu-binfmt start

You might want to add to the services started by default on boot:

root #rc-update add qemu-binfmt


For the systemd-binfmt service, add files containing the desired handler registration strings under /etc/binfmt.d/. For example:

FILE /etc/binfmt.d/qemu-aarch64-static.confSpecifying a binfmt-misc handler for systemd-binfmt
In the above, the eighth byte of the mask has been set to fc (rather than ff), to allow programs like gcc to be executed. See for example Debian bug #799120.

With this done, restart the service:

root #systemctl restart systemd-binfmt

It's started automatically per default, but you might want to confirm it's running properly:

root #systemctl status systemd-binfmt

Setup chroot

Download the desired stage tarball:

root #wget http://arch-stageball

Unpack the tarball:

root #tar -xzvf arch-stageball
root #cd arch-stageball

Install the static qemu into the chroot:

root #ROOT=$PWD/ emerge --usepkgonly --oneshot --nodeps qemu
root #mkdir -p usr/portage

Mount the required directories:

root #mount --bind /usr/portage usr/portage
root #mount --bind /proc proc
root #mount --bind /sys sys
root #mount --bind /dev dev
root #mount --bind /dev/pts dev/pts

Chroot into the environment:

root #chroot . /bin/bash --login

Keep QEMU from being altered within the chroot (if deleted or reinstalled within the chroot, breakage will occur):

root #echo 'EMERGE_DEFAULT_OPTS="$EMERGE_DEFAULT_OPTS --exclude app-emulation/qemu"' >>/etc/portage/make.conf
root #echo app-emulation/qemu >>/var/lib/portage/world

Unmount stuff when not in use:

root #umount usr/portage sys proc dev/pts dev

Sometimes we'll need to pass additional args to QEMU (CPU model), so we'll create a wrapper script (in C) that'll call QEMU with it:

FILE qemu-wrapper.c
 * pass arguments to qemu binary

#include <string.h>
#include <unistd.h>

int main(int argc, char **argv, char **envp) {
	char *newargv[argc + 3];

	newargv[0] = argv[0];
	newargv[1] = "-cpu";
	newargv[2] = "cortex-a8"; /* here you can set the cpu you are building for */

	memcpy(&newargv[3], &argv[1], sizeof(*argv) * (argc -1));
	newargv[argc + 2] = NULL;
	return execve("/usr/bin/qemu-arm", newargv, envp);

Compile the wrapper with:

root #gcc -static qemu-wrapper.c -O3 -s -o qemu-wrapper

Then copy into the chroot. Notice the first example ARM entry in the binfmt_misc section uses this method.

This page is based on a document formerly found on our main website gentoo.org.
The following people contributed to the original document: Mike Frysinger, Ned Ludd, Robin H. Johnson, Alex Tarkovsky, Alexey Shvetsov, Raúl Porcel, Joshua Saddler on April 28, 2013.
They are listed here because wiki history does not allow for any external attribution. If you edit the wiki article, please do not add yourself here; your contributions are recorded on each article's associated history page.