Embedded Handbook/General/Compiling with qemu user chroot
Compiling with QEMU user chroot.
This wiki page explains the required steps needed to be able to chroot into a root filesystem whose platform (e.g. aarch64) is different from the running system (e.g. amd64). Software can then be compiled within the chroot transparently using QEMU emulation.
Prerequsites
Kernel
The host system kernel will need support for miscellaneous binary formats. This is achived via the binfmt_misc module. Add the CONFIG_BINFMT_MISC=m or CONFIG_BINFMT_MISC=y symbols to the host's kernel .config file. If this module is not built already, then the development host will require a reboot after the kernel update and modules_install.
Executable file formats --->
<*> Kernel support for MISC binaries
Package configuration
In order to take advantage of QEMU user mode a few steps are necessary. First the app-emulation/qemu package must be emerged with the right settings. Specifically this means building with USE=static-user and setting QEMU_SOFTMMU_TARGETS and QEMU_USER_TARGETS to include the targets that will be utilized.
See man portage(5) for other ways of doing this:
root #echo "app-emulation/qemu static-user" >> /etc/portage/package.use/qemuActivating the static-user will require supporting libraries to be build with static-libs support:
/etc/portage/package.use/static-libsConfiguring qemu supporting libraries with static-libs# required by app-emulation/qemu::gentoo[static,static-user]
# required by qemu (argument)
dev-libs/glib static-libs
# required by app-emulation/qemu::gentoo[-static,static-user]
# required by qemu (argument)
sys-libs/zlib static-libs
# required by app-emulation/qemu::gentoo[-static,static-user,xattr]
# required by qemu (argument)
sys-apps/attr static-libs
# required by dev-libs/glib::gentoo
# required by app-emulation/qemu::gentoo[-static,static-user]
# required by qemu (argument)
dev-libs/libpcre2 static-libsx
GCC will use huge amount of RAM when LTO is enabled on the system while using the
static-user flag, because of this is recommended to disable LTO while compiling in this configuration or use clang if LTO is required. See bug #883419Tweak the list here to include the necessary target(s). See the output of emerge -pv app-emulation/qemu for the full list:
For Raspberry Pi running a 32-bit stage3 build, be sure to select
QEMU_USER_TARGETS="arm" here. For 64-bit stage3 builds, select aarch64. When in doubt set both.To build all targets:
root #echo 'QEMU_SOFTMMU_TARGETS="alpha aarch64 arm i386 loongarch64 m68k mips mips64 mips64el mipsel ppc ppc64 riscv32 riscv64 s390x sh4 sh4eb sparc sparc64 x86_64"' >> /etc/portage/make.conf
root #echo 'QEMU_USER_TARGETS="alpha aarch64 arm armeb i386 loongarch64 m68k mips mipsel ppc ppc64 ppc64abi32 riscv32 riscv64 s390x sh4 sh4eb sparc sparc32plus sparc64"' >> /etc/portage/make.conf
Alternatively, the USE_EXPAND syntax can be used to keep make.conf free of USE_EXPAND values:
/etc/portage/package.use/qemuConfigure qemu for user chrootapp-emulation/qemu static-user QEMU_SOFTMMU_TARGETS: alpha aarch64 arm i386 loongarch64 m68k mips mips64 mips64el mipsel ppc ppc64 riscv32 riscv64 s390x sh4 sh4eb sparc sparc64 x86_64
app-emulation/qemu QEMU_USER_TARGETS: alpha aarch64 arm armeb i386 loongarch64 m68k mips mipsel ppc ppc64 ppc64abi32 riscv32 riscv64 s390x sh4 sh4eb sparc sparc32plus sparc64 x86_64
Then install the package:
root #emerge --ask --update --newuse --deep app-emulation/qemuAt this point, it is wise to create a binary package for QEMU:
root #quickpkg app-emulation/qemuFinally, for non-root users to use qemu, they will need added to the kvm group. Replace larry in the example below with the appropriate username:
root #gpasswd -a larry kvmAutomatic setup
After this, make sure the binfmt service is (re-)started:
OpenRC
root #rc-service qemu-binfmt startIt may be wise for the services to be started by default on boot:
root #rc-update add qemu-binfmt defaultsystemd
For the systemd-binfmt service, add files containing the desired handler registration strings under /etc/binfmt.d/. Modern versions of qemu ship a binfmt configuration file that supports all binary formats. Simply link it to /etc for binary format support, then skip the following manual file creation steps.
root #ln -s /usr/share/qemu/binfmt.d/qemu.conf /etc/binfmt.d/qemu.confAlternatively, to only make support for aarch64 and arm architectures, separate files can manually be created:
/etc/binfmt.d/qemu-aarch64-static.confSpecifying an aarch64 binfmt-misc handler for systemd-binfmt:aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\xfc\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64:
/etc/binfmt.d/qemu-arm-static.confSpecifying an arm binfmt-misc handler for systemd-binfmt:arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm:
See #Binary format handlers for a list of possible handlers.
In the FileBox above, the eighth byte of the mask has been set to fc (rather than ff), to allow programs like gcc to be executed. See for example Debian bug #799120.
With the file(s) created in /etc/binfmt.d/, systemd will now find them automatically at system boot time. See man 5 binfmt.d for more information.
Since the files were likely created just a few moments ago, it will be necessary to restart the service once to register the binary formats:
root #systemctl restart systemd-binfmtTo confirm the service is running properly after restarting:
root #systemctl status systemd-binfmtManual setup
Mount the binfmt_misc handler if it is not already mounted, then register the format with the kernel via the procfs:
root #[ -d /proc/sys/fs/binfmt_misc ] || modprobe binfmt_misc
root #[ -f /proc/sys/fs/binfmt_misc/register ] || mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc
To register a handler, pipe its format to /proc/sys/fs/binfmt_misc/register, for example
root #echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm:' > /proc/sys/fs/binfmt_misc/registersee #Binary format handlers for a list of possible handlers
Do not register a handler that matches the host machine (in this example, registers for x86 and AMD64 has been excluded since the host machine).
Chrooting
To be able to chroot into a system of a different platform (e.g. aarch64 while using an amd64 system), mounted in e.g. /mnt/gentoo.
- Copy over the relevant QEMU binary built at #Package configuration section (e.g. /usr/bin/qemu-aarch64) to /mnt/gentoo/usr/bin/ (thus the importance of a statically linked binary).
- Chroot into the environment using sys-apps/arch-chroot
root #arch-chroot /mnt/gentoo
Advanced Tips
Sometimes we'll need to pass additional args to QEMU (CPU model), so we'll create a wrapper script (in C) that'll call QEMU with it:
qemu-wrapper.c/*
* Pass arguments to qemu binary
*/
#include <string.h>
#include <unistd.h>
int main(int argc, char **argv, char **envp) {
char *newargv[argc + 3];
newargv[0] = argv[0];
newargv[1] = "-cpu";
newargv[2] = "cortex-a8"; /* here you can set the cpu you are building for */
memcpy(&newargv[3], &argv[1], sizeof(*argv) * (argc -1));
newargv[argc + 2] = NULL;
return execve("/usr/bin/qemu-arm", newargv, envp);
}
Compile the wrapper with:
root #gcc -static qemu-wrapper.c -O3 -s -o qemu-wrapperThen copy into the chroot. Notice the first example ARM entry in the binfmt_misc section uses this method.
Caveats
Currently QEMU does not support the pid-sandbox (bug #703278) and network-sandbox (bug #703276) features of Portage.
This issue can be worked around by setting the FEATURES environment variable before each call to emerge and other Portage tools in the chroot:
root #env FEATURES="-pid-sandbox -network-sandbox" emerge
root #env FEATURES="-pid-sandbox -network-sandbox" equery
Alternatively, modify Portage's FEATURES variable within the chroot.
/etc/portage/make.confFEATURES="-pid-sandbox -network-sandbox"
Binary format handlers
| Architecture |
Binfmt Handler String |
|---|---|
| AARCH64 | :aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\xfc\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64:
|
| ARM | :arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm:
|
| ARMeb | :armeb:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-armeb:
|
| Alpha | :alpha:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x26\x90:\xff\xff\xff\xff\xff\xfe\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-alpha:
|
| LOONGARCH64 | :loongarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x02\x01:\xff\xff\xff\xff\xff\xff\xff\xfc\x00\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-system-loongarch64:
|
| M68K | :m68k:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x04:\xff\xff\xff\xff\xff\xff\xfe\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-m68k:
|
| MIPS | :mips:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-mips:
|
| MIPS64 | :mips64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xf3\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-mips64:
|
| MIPSel | :mipsel:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-mipsel:
|
| MIPS N32 | :mipsn32:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-mipsn32:
|
| MIPSel N32 | :mipseln32:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-mipseln32:
|
| PPC | :ppc:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x14:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-ppc:
|
| PPC64 | :ppc64:M::\x7fELF\x02\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x15:\xff\xff\xff\xff\xff\xff\xff\xfc\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-ppc64:
|
| RISCV64 | :riscv64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xf3\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-riscv64:
|
| SH4 | :sh4:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x2a\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfb\xff\xff\xff:/usr/bin/qemu-sh4:
|
| SH4eb | :sh4eb:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x2a:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-sh4eb:
|
| SPARC | :sparc:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x02:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-sparc:
|
This page is based on a document formerly found on our main website gentoo.org.
The following people contributed to the original document: Mike Frysinger, Ned Ludd, Robin H. Johnson, Alex Tarkovsky, Alexey Shvetsov, Raúl Porcel, Joshua Saddler on April 28, 2013.
They are listed here because wiki history does not allow for any external attribution. If you edit the wiki article, please do not add yourself here; your contributions are recorded on each article's associated history page.