Embedded Handbook/General/Full

From Gentoo Wiki
Jump to:navigation Jump to:search


Introduction

Cross development has traditionally been a black art, requiring a lot of research, trial and error, and perseverance. Intrepid developers face a shortage of documentation and the lack of mature, comprehensive open source toolkits for multi-platform cross development. Ongoing work by the Embedded or Toolchain projects, and other contributors is yielding a Gentoo-based development platform that greatly simplifies cross development.

The toolchain

The term "toolchain" refers to the collection of packages used to build up a system (the "tools" which are used in the "chain" of events to take some input and produce some output). It is a loose definition in terms of what packages exactly are considered part of the toolchain, but for the sake of keeping things simple, we will consider the components that are needed to compile code into something fun and usable.

Your typical toolchain is therefore composed of the following:

sys-devel/binutils
Essential utilities for handling binaries (includes assembler and linker).
sys-devel/gcc
The GNU Compiler Collection (the C and C++ compiler).
sys-libs/glibc, sys-libs/uclibc-ng, or sys-libs/newlib
The system C library.
sys-kernel/linux-headers
Kernel headers needed by the system C library.
sys-devel/gdb
The GNU debugger.

All proper Gentoo systems have a toolchain installed as part of the base system. This toolchain is configured to build binaries native to its host platform.

In order to build binaries on the host system for a non-native platform you'll need a special toolchain - a so-called cross toolchain - which can target that particular platform. Gentoo provides a simple but powerful tool called crossdev for this purpose. Crossdev can build and install arbitrary GCC-supported cross toolchains on the host system, and because Gentoo installs toolchain files into target-specific directories the toolchains built by crossdev will not interfere with the host's native toolchain.

Toolchain tuples

All toolchains have a prefix (think CHOST). More details on that can be found in the system tuples article.

Environment variables

Certain environment variables used by the Gentoo toolchain and Portage can thoroughly confuse developers inexperienced with cross development. The following table explains some tricky variables and provides sample values based on the cross development examples presented in this guide. See More terminology and variables (below) for more unusual variables and related concepts.

Variable name Meaning when building cross-toolchain Meaning when building cross-binaries
CBUILD Platform you are building on Platform you are building on
CHOST Platform the cross-toolchain will run on Platform the binaries built by cross-toolchain will run on
CTARGET Platform the binaries built by cross-toolchain will run on Platform the binaries built by cross-toolchain will run on. Redundant, but there's no harm in setting this, and a few binaries do like it.
ROOT Path to the virtual root (/) you are installing into
PORTAGE_CONFIGROOT Path to the virtual root (/) Portage can find its config files (like /etc/make.conf)

Say we have an AMD64 desktop as our normal Gentoo machine and we have an ARM PDA we wanted to develop for, the above table would look like:

Variable name Value for building cross-toolchain Value for building cross-binaries
CBUILD x86_64-pc-linux-gnu x86_64-pc-linux-gnu
CHOST x86_64-pc-linux-gnu arm-unknown-linux-gnu
CTARGET arm-unknown-linux-gnu Not set.
ROOT Not set - defaults to / /path/where/you/install
PORTAGE_CONFIGROOT Not set - defaults to / /path/where/your/portage/env/for/arm/pda/is

More terminology and variables

canadian cross
The process of building a cross-compiler which will run on a different machine from the one it was compiled on (CBUILD != CHOST && CHOST != CTARGET)
sysroot
The system root is where all the cross-compiler libraries and headers are installed. In other words, every library and header the cross-compiler needs to generate cross-compiled binaries are put into this directory. In theory, this means the toolchain is good enough. In practice, the cross-compiler often wants some of a packages's library dependencies, such as ncurses, installed into sysroot first.
hardfloat
The system has a hardware Floating Point Unit (FPU) to handle floating point math
softfloat
The system lacks a hardware FPU so all floating point operations are approximated with fixed point math
PIE
Position Independent Executable (-fPIE -pie)
PIC
Position Independent Code (-fPIC)
CRT
C run time


Creating a cross-compiler

The first thing users should know about building a toolchain is that some versions of toolchain components refuse to work together. Exactly which combinations are problematic is a matter that's constantly in flux as the Gentoo ebuild repository evolves. The only reliable way to determine what works is to run crossdev, adjusting individual component versions as necessary, until crossdev completes the toolchain build successfully. Even then, the cross toolchain may build binaries which break on the target system. Only through trial and error and patience will be needed to arrive at a favorable combination of all factors.

Users do not have to worry about the cross-compiler interfering with the native build system. All of the toolchain packages are designed such that they are isolated from each other based on the target. This way cross-compilers can be installed for desired architecture(s) without breaking the rest of the system.

crossdev

Intro

Generating a cross-compiler by hand is a long and painful process. This is why it has been fully integrated into Gentoo! A command-line front-end called crossdev will run emerge with all of the proper environment variables and install all the right packages to generate arbitrary cross-compilers based on the need of the user.

First, install Crossdev:

root #emerge --ask sys-devel/crossdev

Consider installing the unstable version of crossdev to get all the latest fixes.

Note
Those upgrading from older versions of crossdev, and have crossdev-wrappers installed, need to depclean crossdev-wrappers first. The existing cross-toolchains will remain intact.

Only basic usage of crossdev is covered here, but crossdev can customize the process fairly well for most needs. Run crossdev --help to get some ideas on how to use crossdev. Here are some common usage options:

crossdev --g [gcc version] --l [(g)libc version] --b [binutils version] --k [kernel headers version] -P -v -t [tuple]
crossdev -S -P -v -t [tuple]

Installing

The first step is to select the proper tuple for the target. Here, it will be assumed that a cross-compiler for the SH4 (SuperH) processor with glibc running on Linux is desired to be built by the user. This action will be performed on a PowerPC machine. Generate a SH4 cross-compiler:

root #crossdev --target sh4-unknown-linux-gnu
-----------------------------------------------------------------------------------------------------
 * Host Portage ARCH:     ppc
 * Target Portage ARCH:   sh
 * Target System:         sh4-unknown-linux-gnu
 * Stage:                 4 (C/C++ compiler)

 * binutils:              binutils-[latest]
 * gcc:                   gcc-[latest]
 * headers:               linux-headers-[latest]
 * libc:                  glibc-[latest]

 * PORTDIR_OVERLAY:       /var/db/repos/local
 * PORT_LOGDIR:           /var/log/portage
 * PKGDIR:                /usr/portage/packages/powerpc-unknown-linux-gnu/cross/sh4-unknown-linux-gnu
 * PORTAGE_TMPDIR:        /var/tmp/cross/sh4-unknown-linux-gnu
  _  -  ~  -  _  -  ~  -  _  -  ~  -  _  -  ~  -  _  -  ~  -  _  -  ~  -  _  -  ~  -  _  -  ~  -  _  
 * Forcing the latest versions of {binutils,gcc}-config/gnuconfig ...                          [ ok ]
 * Log: /var/log/portage/cross-sh4-unknown-linux-gnu-binutils.log
 * Emerging cross-binutils ...                                                                 [ ok ]
 * Log: /var/log/portage/cross-sh4-unknown-linux-gnu-gcc-stage1.log
 * Emerging cross-gcc-stage1 ...                                                               [ ok ]
 * Log: /var/log/portage/cross-sh4-unknown-linux-gnu-linux-headers.log
 * Emerging cross-linux-headers ...                                                            [ ok ]
 * Log: /var/log/portage/cross-sh4-unknown-linux-gnu-glibc.log
 * Emerging cross-glibc ...                                                                    [ ok ]
 * Log: /var/log/portage/cross-sh4-unknown-linux-gnu-gcc-stage2.log
 * Emerging cross-gcc-stage2 ...                                                               [ ok ]
Note
At the moment it is not possible to set PORTAGE_CONFIGROOT before calling crossdev to a folder set to the architecture for the target. A custom config must be used. To use arch specific USE flags, like altivec in a non-PowerPC architecture, unmask the use flag in /var/db/repos/gentoo/profiles/base/use.mask, or temporarily change the system profile.

Quick test

If everything goes as planned, a shiny new compiler should now be present on the machine. Give it a spin!

Use the SH4 cross-compiler:

user $sh4-unknown-linux-gnu-gcc --version
sh4-unknown-linux-gnu-gcc (GCC) 4.2.0 (Gentoo 4.2.0 p1.4)
Copyright (C) 2007 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
user $echo 'int main(){return 0;}' > sh4-test.c
user $sh4-unknown-linux-gnu-gcc -Wall sh4-test.c -o sh4-test
user $file sh4-test
sh4-test: ELF 32-bit LSB executable, Renesas SH, version 1 (SYSV), for GNU/Linux 2.6.9, dynamically linked (uses shared libs), not stripped

If the crossdev command failed, the log file may be reviewed to see if the problem is local. If unable to fix the issue, a bug may be filed in Bugzilla.

Tuples

To find out which tuple should be used, look over the output from the following command:

root #crossdev -t help

There should now be a newly compiled cross-compiler in the sysroot at /usr/${CTARGET}/. It's a good idea to create pre-built binary packages so as to not end up waiting another two to three hours every time this toolchain should be reinstalled.

Create binpkgs

Note
The default value for creating binary packages is xpak. For more information on managing binary packages please refer to the Binary package guide.
root #quickpkg --include-unmodified-config=y cross-sh4-unknown-linux-gnu/gcc
root #quickpkg --include-unmodified-config=y cross-sh4-unknown-linux-gnu/glibc
root #quickpkg --include-unmodified-config=y cross-sh4-unknown-linux-gnu/binutils
root #quickpkg --include-unmodified-config=y cross-sh4-unknown-linux-gnu/linux-headers

If the quickpkg command warns about excluded files, please follow its prompts to include all files.

In the future the sysroot can be reinstalled by executing the following simple Portage command:

root #emerge -k cross-sh4-unknown-linux-gnu/gcc cross-sh4-unknown-linux-gnu/glibc cross-sh4-unknown-linux-gnu/binutils cross-sh4-unknown-linux-gnu/linux-headers
Tip
For the sake of solidarity (to prevent inferences), the definition and subsequent explanation of sysroot are located in the Introduction and outlined under Cross-compiler internals below; respectively.

Uninstalling

To uninstall a toolchain, simply use the --clean option. If the sysroot was modified by hand, there will be a prompt to delete every file inside, so it is possible to prepend yes | to this command if there is no doubt about what can be deleted:

Uninstall the SH4 cross-compiler:

root #crossdev --clean sh4-unknown-linux-gnu

Deleting any and all files in the /usr/${CTARGET}/ directory should be completely safe.

Cross-compiler internals

Tip
This section is included for posterity and in the hopes that others will find it useful. The target audience is all users who (due to curiosity, naïveté, general misconceptions, and/or their current inability to comprehend this warning) really really want to create their own cross compiler with binutils/(glibc/musl/some other libc)/gcc all by themselves. This section is a synopsys, and lacks the relevant documentation that would prevent the myriad of build failures that may be encountered. If such help is required, see the Beyond section in the handbook for some pointers. Please do not ask for any support at all for this.
Warning
If still reading, it is a very good idea to check out the crosstool project (again refer to the Beyond section) as that provides a distribution independent method for generating cross-compilers. While imperfect, it is certainly the best (and really only) option out there for creating cross-compilers.

Overview

There are generally two ways to build a cross-compiler. The "accepted" way, and the cheater's shortcut.

The current "accepted" way is:

  1. binutils
  2. kernel headers
  3. libc headers
  4. gcc stage1 (c-only)
  5. libc
  6. gcc stage2 (c/c++/etc...)

The cheater's shortcut is:

  1. binutils
  2. kernel headers
  3. gcc stage1 (c-only)
  4. libc
  5. gcc stage2 (c/c++/etc...)

The reason people are keen on the shortcut is that the libc headers step tends to take quite a while, especially on slower machines. It can also be kind of a pain to setup kernel/libc headers without a usable cross compiler. Note, though, that if help with cross-compilers is sought, upstream projects will not want to help if the shortcut was taken.

Also note that the shortcut requires the gcc stage1 to be "crippled". Since building without headers, the sysroot option cannot be enabled nor proper gcc libs can be built. This is okay if the only thing that is being used in the stage1 is building the C library and a kernel, but beyond that, a nice sysroot-based compiler is needed.

The "accepted" way is described below as the steps are pretty much the same. Some extra patches are needed for gcc in order to take the shortcut.

Sysroot

The cross-compiling will be done using the sysroot method. But what does the sysroot do?

The sysroot tells GCC to consider dir as the root of a tree that contains (a subset of) the root filesystem of the target operating system. Target system headers, libraries and run-time object files will be searched in there.

The top level directory is commonly rooted in /usr/$CTARGET

CODE A typical sysroot layout
|-- bin/
|-- lib/            critical runtime libs (libc/ldso/etc...)
`-- usr/
    |-- include/    development headers
    |   |-- linux/      like the linux kernel
    |   `-- asm/        like the arch-specific
    `-- lib/        non critical runtime libs / development libs

As can be seen, it's just like the directory setup in / but in /usr/$CTARGET. This setup is of course not an accident but designed on purpose so applications/libraries can be easily migrated out of /usr/$CTARGET and into / on the target board. If desired, /usr/$CTARGET could be used as a quick NFS root!

Note
The old style of cross-compilers was to use --prefix=/usr/$CTARGET. If those binutils/gcc versions are used that predate sysroot support, it may be needed to do just this. It will not be documented because (1) old/crusty/busted versions should not be used and (2) it's quite a huge pain compared to sysroot.

Binutils

Grab the latest binutils tarball and unpack it.

The --disable-werror option is to prevent binutils from aborting the compile due to warnings. Great feature for developers, but a pain for users. Configure and build binutils:

root #make
root #make install DESTDIR=$PWD/install-root

The reason of install into the localdir is the crap that doesn't belong can be removed. For example, a normal install will give /usr/lib/libiberty.a which doesn't belong in the host /usr/lib. So clean out stuff first:

root #rm -rf install-root/usr/{info,lib,man,share}

And install what's left:

root #cp -a install-root/* /

Kernel headers

Grab the latest Linux tarball and unpack it. There are two ways of installing the kernel headers: sanitized and unsanitized. It is clear that the former is better (but it requires a recent version of the Linux kernel), but both will be quickly covered.

Note
Clearly the $ARCH variable need to be replaced with a value that makes sense for needed platform.

Building and install the unsanitized headers:

root #yes "" | make ARCH=$ARCH oldconfig prepare
root #mkdir -p /usr/$CTARGET/usr/include
root #cp -a include/linux include/asm-generic /usr/$CTARGET/usr/include/
root #cp -a include/asm-$ARCH /usr/$CTARGET/usr/include/asm

Build and install the sanitized headers:

root #make ARCH=$ARCH headers_install INSTALL_HDR_PATH=/usr/$CTARGET/usr

System libc headers

Grab the latest glibc tarball and unpack it. Glibc is picky, so compilation have to be done in a directory separate from the source code. Build and install the glibc headers:

root #mkdir build
root #cd build
root #../configure --host=$CTARGET --prefix=/usr --with-headers=/usr/$CTARGET/usr/include --without-cvs --disable-sanity-checks
root #make -k install-headers install_root=/usr/$CTARGET

glibc can be awkward sometimes, so have to do a few things by hand:

root #mkdir -p /usr/$CTARGET/usr/include/gnu
root #touch /usr/$CTARGET/usr/include/gnu/stubs.h
root #cp bits/stdio_lim.h /usr/$CTARGET/usr/include/bits/

GCC stage 1 (C only)

At first, help gcc find the current libc headers:

root #ln -s usr/include /usr/$CTARGET/sys-include

Then grab the latest gcc tarball and unpack it:

root #mkdir build
root #cd build
root #make
root #make install DESTDIR=$PWD/install-root

Same as binutils, gcc leaves some stuff behind that doesn't wanted. Clean the gcc stage 1:

user $rm -rf install-root/usr/{info,include,lib/libiberty.a,man,share}

System libc

Remove the old glibc build directory and recreate it:

root #rm -rf build
root #mkdir build
root #cd build
root #../configure --host=$CTARGET --prefix=/usr --without-cvs
root #make
root #make install install_root=/usr/$CTARGET

GCC stage 2 (all frontends)

A full GCC can be built up now. Whichever compiler frontends are preferred can be selected; C/C++ will just be done for simplicity. Build and install the gcc stage 2:

root #./configure --target=$CTARGET --prefix=/usr --with-sysroot=/usr/$CTARGET --enable-languages=c,c++ --enable-shared --disable-checking --disable-werror
root #make
root #make install

Core runtime files

There are many random core runtime files that people wonder what they may be for. Let's explain:

Files provided by sys-libs/glibc
File Purpose
crt0.o Older style of the initial runtime code. No one generates this anymore.
crt1.o Newer style of the initial runtime code. Contains the _start symbol which sets up the env with argc/argv/libc _init/libc _fini before jumping to the libc main. glibc calls this file 'start.S'.
crti.o Defines the function prolog; _init in the .init section and _fini in the .fini section. glibc calls this 'initfini.c'.
crtn.o Defines the function epilog. glibc calls this 'initfini.c'.
Scrt1.o Used in place of crt1.o when generating PIEs.
gcrt1.o Used in place of crt1.o when generating code with profiling information. Compile with -pg. Produces output suitable for the gprof util.
Mcrt1.o Like gcrt1.o, but is used with the prof utility. glibc installs this as a dummy file as it's useless on linux systems.
Files provided by sys-devel/gcc
File Purpose
crtbegin.o GCC uses this to find the start of the constructors.
crtbeginS.o Used in place of crtbegin.o when generating shared objects/PIEs.
crtbeginT.o Used in place of crtbegin.o when generating static executables.
crtend.o GCC uses this to find the start of the destructors.
crtendS.o Used in place of crtend.o when generating shared objects/PIEs.

The general linking order:

crt1.o crti.o crtbegin.o [-L paths] [user objects] [gcc libs] [C libs] [gcc libs] crtend.o crtn.o

External resources

GCC Cross-Compiler


Cross-compiling with Portage

Variables

There are a few important variables that will be used throughout this section:

Variable name Meaning
CBUILD The platform being built on.
CHOST The platform being compiled on.
ROOT The virtual / being installed into.
PORTAGE_CONFIGROOT The virtual / Portage can find its config files (like make.conf).

These variables can be set by hand, but that obviously gets tedious very quickly. A better idea is to stick these into a shell script to avoid typing them each time.

Filesystem setup

Cross-compiling a system generally involves two directory structures. The differences between them can be difficult to understand, but they are important concepts to cross-compiling.

The first directory structure to consider is the sysroot. Please read the Introduction chapter for the definition of sysroot.

The second directory structure is the real root. This one is much simpler to understand: it's where the bootable stage3-like installation of Gentoo is located. This installation can then be copied to the target device.

The common convention is to use the /usr/${CTARGET}/ directory as the sysroot since the include/library directories in this tree are already encoded into the gcc cross-compiler for searching. Another directory could be used and then custom -I/-L paths could be added to the CPPFLAGS/LDFLAGS, but this has historically proven to be problematic in enough corner cases to be discouraged for practical use. In the embedded handbook, it will be assumed the sysroot is being used as the development ROOT.

A much slimmer/trimmed-down setup will be needed for the runtime system. The files removed from a normal installed package are the reason why this tree is not suitable for compiling against. If binary packages are built while installing into the sysroot, then those binary packages can be used in conjunction with the INSTALL_MASK variable to trim out most things. More information can be found in the man make.conf.

Intro: crossdev's wrappers

Simple wrapper scripts will be used to setup the environment variables to point to the appropriate locations, enabling cross-compilation using emerge. PORTAGE_CONFIGROOT and ROOT both point to the SYSROOT.

root #emerge --ask sys-devel/crossdev
Warning
crossdev's wrappers automatically configure the CBUILD and CHOST variables. Do not set them by hand in the environment!
Note
Before any cross-emerge can be started, emerge-wrapper --init needs to be run. Instructions printed by emerge-wrapper should be followed before starting cross-emerge.

These tools can be used for both installing into the development root (sysroot) and into the runtime root. For the latter, the --root option should be specified. For example if an armv4tl-softfloat-linux-gnueabi toolchain has been merged via crossdev, then command can then be invoked like a normal emerge. But using the ctarget prefix:

root #armv4tl-softfloat-linux-gnueabi-emerge pkg0 pkg1 pkg2

By default these wrappers use the --root-deps=rdeps option to avoid the host dependencies from being pulled into the deptree. This can lead to incomplete deptrees. Therefore the --root-deps option can be used alone to see the full dependency graph.

By default crossdev will link to the generic embedded profile. This is done to simplify things, but the user may wish to use a more advanced targeted profile. The profile symlink can be updated in order to do that.

root #ln -s /var/db/repos/gentoo/profiles/default/linux/arm/17.0 ${SYSROOT}/etc/portage/make.profile

To change settings (such as USE flags) for the target system, edit the standard Portage config files:

root #${EDITOR} ${SYSROOT}/etc/portage/make.conf

Sometimes there are some additional, cross-compile-incompatible tests that must be overridden for configure scripts. To make this possible, crossdev exports a few variables to force the test to get the answer it should receive. This helps prevent bloat in packages which add local functions to workaround issues it assumes the target system has because it could not run the test while cross-compiling (because, of course, it's not running on the target system). From time-to-time additional variables may need to be added to these files in /usr/share/crossdev/include/site/ in order to get a package to compile. To figure out the variable that needs added, it is often as simple as grepping the configure script for the autoconf variable (which often begins with ac_) and adding it to the appropriate target file. This becomes easier with ebuild development experience.

Uninstall

If uninstalling and deleting the work is desired, the sysroot tree can be safely removed without affecting any native packages. Refer to the "Uninstalling" section in the crossdev guide.


Cross-compiling the kernel

Sources

The relevant kernel sources should be installed first. A kernel sources package can be quickly emerged from the Gentoo ebuild repository or fetch the latest sources from kernel.org. The method for actually compiling the kernel is all the same.

The kernel should be installed into the sysroot so if desired to cross-compile packages which include kernel modules, the process will be transparent. Otherwise, the actual location where the kernel is built does not matter. Some people build all their kernels in /usr/src/ for example.

Setup cross-compiling

There are two fundamental variables that the kernel uses to select the target architecture. Normally these values are guessed based on the build environment, but of course that environment here does not match the target embedded system, so they will need to be overriden. The variables in question are ARCH and CROSS_COMPILE. The default values for both are found in the top-level Makefile and the values of both may be overridden on the command line.

The ARCH variable refers to the architecture that the kernel is being built for, as recognized by the kernel itself. So while portage and other people may use "x86", the kernel uses "i386". The arch/ subdirectory should be quickly checked to determine which architecture is to be used.

Hopefully the CROSS_COMPILE variable is pretty self-explanatory. It should be set to the prefix of the toolchain (including the trailing dash "-"). So if the toolchain is invoked as say x86_64-pc-linux-gnu-gcc, the trailing gcc should be chopped off and that's what should be used: x86_64-pc-linux-gnu-.

There is an additional variable, INSTALL_MOD_PATH, which defines where the /lib directory will be created, and all the modules stored. The kernel sources do not have to be transferred to the target device, but if any modules are built, this directory will be needed.

There are really two ways in which the system can be set up. The toplevel Makefile can be modified or the relevant variables can be overridden on the command line. Both methods are acceptable, so both will be covered. One of the following options can be chosen.

FILE MakefileThe vanilla Makefile
ARCH            ?= $(SUBARCH)
CROSS_COMPILE   ?=
FILE MakefileSet the ARCH and CROSS_COMPILE default values
ARCH            ?= arm
CROSS_COMPILE   ?= arm-unknown-linux-gnu-

Overriding on the command-line (instead of in the Makefile) would look something like this:

root #make ARCH=arm CROSS_COMPILE=arm-unknown-linux-gnu-

A little helper script can be used if it is necessary to switch between different kernel trees at the same time. The script will be referred to as xkmake:

FILE xkmake
#!/bin/sh
exec make ARCH="arm" CROSS_COMPILE="arm-unknown-linux-gnu-" INSTALL_MOD_PATH="${SYSROOT}" "$@"

Now, when building a kernel or performing any other action, xkmake should be executed instead of make.

Configure and compile

At this point, configuring and compiling the kernel is the same as any other kernel, so it won't be discussed in depth here. There are numerous HOWTOs and guides available that can cover the subject in much greater detail.

root #cd "${SYSROOT}/usr/src/linux"
root #xkmake menuconfig
root #xkmake


Cross compiling software with a QEMU user chroot.


Prerequsites

Kernel

The host system kernel will need support for miscellaneous binary formats. This is achived via the binfmt_misc module. Add the CONFIG_BINFMT_MISC=m or CONFIG_BINFMT_MISC=y symbols to the host's kernel .config file. If this module is not built already, then the development host will require a reboot after the kernel update and modules_install.

KERNEL Enable CONFIG_BINFMT_MISC
Executable file formats  --->
  <*> Kernel support for MISC binaries

Package configuration

In order to take advantage of QEMU user mode a few steps are necessary. First the app-emulation/qemu package must be emerged with the right settings. Specifically this means building with USE=static-user and setting QEMU_SOFTMMU_TARGETS and QEMU_USER_TARGETS to include the targets that will be utilized.

See man portage(5) for other ways of doing this:

root #echo "app-emulation/qemu static-user" >> /etc/portage/package.use/qemu

Activating the static-user will require supporting libraries to be build with static-libs support:

FILE /etc/portage/package.use/static-libsConfiguring qemu supporting libraries with static-libs
# required by app-emulation/qemu::gentoo[static,static-user]
# required by qemu (argument)
dev-libs/glib static-libs
# required by app-emulation/qemu::gentoo[-static,static-user]
# required by qemu (argument)
sys-libs/zlib static-libs
# required by app-emulation/qemu::gentoo[-static,static-user,xattr]
# required by qemu (argument)
sys-apps/attr static-libs
# required by dev-libs/glib::gentoo
# required by app-emulation/qemu::gentoo[-static,static-user]
# required by qemu (argument)
dev-libs/libpcre2 static-libsx
Note
GCC will use huge amount of RAM when LTO is enabled on the system while using the static-user flag, because of this is recommended to disable LTO while compiling in this configuration or use clang if LTO is required. See bug #883419

Tweak the list here to include the necessary target(s). See the output of emerge -pv app-emulation/qemu for the full list:

Important
For Raspberry Pi running a 32-bit stage3 build, be sure to select QEMU_USER_TARGETS="arm" here. For 64-bit stage3 builds, select aarch64. When in doubt set both.

To build all targets:

root #echo 'QEMU_SOFTMMU_TARGETS="alpha aarch64 arm i386 m68k mips mips64 mips64el mipsel ppc ppc64 riscv32 riscv64 s390x sh4 sh4eb sparc sparc64 x86_64"' >> /etc/portage/make.conf
root #echo 'QEMU_USER_TARGETS="alpha aarch64 arm armeb i386 m68k mips mipsel ppc ppc64 ppc64abi32 riscv32 riscv64 s390x sh4 sh4eb sparc sparc32plus sparc64"' >> /etc/portage/make.conf

Alternatively, the USE_EXPAND syntax can be used to keep make.conf free of USE_EXPAND values:

FILE /etc/portage/package.use/qemuConfigure qemu for user chroot
app-emulation/qemu static-user QEMU_SOFTMMU_TARGETS: alpha aarch64 arm i386 m68k mips mips64 mips64el mipsel ppc ppc64 riscv32 riscv64 s390x sh4 sh4eb sparc sparc64 x86_64
app-emulation/qemu QEMU_USER_TARGETS: alpha aarch64 arm armeb i386 m68k mips mipsel ppc ppc64 ppc64abi32 riscv32 riscv64 s390x sh4 sh4eb sparc sparc32plus sparc64 x86_64

Then install the package:

root #emerge --ask --update --newuse --deep app-emulation/qemu

At this point, it is wise to create a binary package for QEMU:

root #quickpkg app-emulation/qemu

Finally, for non-root users to use qemu, they will need added to the kvm group. Replace larry in the example below with the appropriate username:

root #gpasswd -a larry kvm

Register binary format handlers

Mount the binfmt_misc handler if it is not already mounted, then register the format with the kernel via the procfs:

root #[ -d /proc/sys/fs/binfmt_misc ] || modprobe binfmt_misc
root #[ -f /proc/sys/fs/binfmt_misc/register ] || mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc

Do not register a handler that matches the host machine (in this example, registers for x86 and AMD64 has been excluded since the host machine).

32-bit registers:

root #echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm:' > /proc/sys/fs/binfmt_misc/register
root #echo ':armeb:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-armeb:' > /proc/sys/fs/binfmt_misc/register
root #echo ':alpha:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x26\x90:\xff\xff\xff\xff\xff\xfe\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-alpha:' > /proc/sys/fs/binfmt_misc/register
root #echo ':mips:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-mips:' > /proc/sys/fs/binfmt_misc/register
root #echo ':mipsel:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-mipsel:' > /proc/sys/fs/binfmt_misc/register
root #echo ':ppc:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x14:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-ppc:' > /proc/sys/fs/binfmt_misc/register
root #echo ':sh4:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x2a\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfb\xff\xff\xff:/usr/bin/qemu-sh4:' >/proc/sys/fs/binfmt_misc/register
root #echo ':sh4eb:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x2a:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-sh4eb:' >/proc/sys/fs/binfmt_misc/register
root #echo ':sparc:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x02:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-sparc:' > /proc/sys/fs/binfmt_misc/register
root #echo ':m68k:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x04:\xff\xff\xff\xff\xff\xff\xfe\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-m68k:' > /proc/sys/fs/binfmt_misc/register

Add 64-bit registers:

root #echo ':aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\xfc\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64:' > /proc/sys/fs/binfmt_misc/register
root #echo ':ppc64:M::\x7fELF\x02\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x15:\xff\xff\xff\xff\xff\xff\xff\xfc\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-ppc64:' > /proc/sys/fs/binfmt_misc/register
root #echo ':riscv64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xf3\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-riscv64:' > /proc/sys/fs/binfmt_misc/register
root #echo ':mips64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xf3\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-mips64' > /proc/sys/fs/binfmt_misc/register

Services

After this, make sure the binfmt service is (re-)started:

OpenRC

root #rc-service qemu-binfmt start

It may be wise for the services to be started by default on boot:

root #rc-update add qemu-binfmt default

systemd

For the systemd-binfmt service, add files containing the desired handler registration strings under /etc/binfmt.d/. Modern versions of qemu ship a binfmt configuration file that supports all binary formats. Simply link it to /etc for binary format support, then skip the following manual file creation steps.

root #ln -s /usr/share/qemu/binfmt.d/qemu.conf /etc/binfmt.d/qemu.conf

Alternatively, to only make support for aarch64 and arm architectures, files separate files can manually be created:

FILE /etc/binfmt.d/qemu-aarch64-static.confSpecifying an aarch64 binfmt-misc handler for systemd-binfmt
:aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\xfc\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64:
FILE /etc/binfmt.d/qemu-arm-static.confSpecifying an arm binfmt-misc handler for systemd-binfmt
:arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm:
Note
In the FileBox above, the eighth byte of the mask has been set to fc (rather than ff), to allow programs like gcc to be executed. See for example Debian bug #799120.

With the file(s) created in /etc/binfmt.d/, systemd will now find them automatically at system boot time. See man 5 binfmt.d for more information.

Since the files were likely created just a few moments ago, it will be necessary to restart the service once to register the binary formats:

root #systemctl restart systemd-binfmt

To confirm the service is running properly after restarting:

root #systemctl status systemd-binfmt

Setup chroot

Download the desired stage tarball:

root #wget http://arch-stageball

Unpack the tarball:

root #tar -xzvf arch-stageball
root #cd arch-stageball

Install the static QEMU into the chroot:

root #ROOT=$PWD/ emerge --usepkgonly --oneshot --nodeps qemu
root #mkdir -p var/db/repos/gentoo

Mount the required directories:

root #mount --bind /var/db/repos/gentoo var/db/repos/gentoo
root #mount --bind /proc proc
root #mount --bind /sys sys
root #mount --bind /dev dev
root #mount --bind /dev/pts dev/pts

Chroot into the environment:

root #chroot . /bin/bash --login

Keep QEMU from being altered within the chroot (if deleted or reinstalled within the chroot, breakage will occur):

root #echo 'EMERGE_DEFAULT_OPTS="$EMERGE_DEFAULT_OPTS --exclude app-emulation/qemu"' >> /etc/portage/make.conf
root #echo app-emulation/qemu >> /var/lib/portage/world

Unmount the various file systems when they are no longer in use:

root #umount var/db/repos/gentoo sys proc dev/pts dev

Sometimes we'll need to pass additional args to QEMU (CPU model), so we'll create a wrapper script (in C) that'll call QEMU with it:

FILE qemu-wrapper.c
/*
 * Pass arguments to qemu binary
 */

#include <string.h>
#include <unistd.h>

int main(int argc, char **argv, char **envp) {
	char *newargv[argc + 3];

	newargv[0] = argv[0];
	newargv[1] = "-cpu";
	newargv[2] = "cortex-a8"; /* here you can set the cpu you are building for */

	memcpy(&newargv[3], &argv[1], sizeof(*argv) * (argc -1));
	newargv[argc + 2] = NULL;
	return execve("/usr/bin/qemu-arm", newargv, envp);
}

Compile the wrapper with:

root #gcc -static qemu-wrapper.c -O3 -s -o qemu-wrapper

Then copy into the chroot. Notice the first example ARM entry in the binfmt_misc section uses this method.

Caveat

Currently QEMU does not support the pid-sandbox (bug #703278) and network-sandbox (bug #703276) features of Portage.

This issue can be worked around by setting the FEATURES environment variable before each call to emerge and other Portage tools in the chroot:

root #env FEATURES="-pid-sandbox -network-sandbox" emerge
root #env FEATURES="-pid-sandbox -network-sandbox" equery

Alternatively, modify Portage's FEATURES variable within the chroot.

FILE /etc/portage/make.conf
FEATURES="-pid-sandbox -network-sandbox"


Frequently asked questions

I get "configure: error: C compiler cannot create executables"

This is a generic error and can be caused by just about anything. The test is pretty simple: can the requested compiler create an executable? However, this relies on many things being correct: the toolchain itself being completely sane, the compiler and compiler flags being appropriate, your environment set up properly, etc... The only way to find out the real source of the problem is to open up the generated config.log file and scroll down to where this test is run and see what exactly the error message is that the toolchain is spitting out.

"epatch" always fails in newly compiled system

The bash package does not properly cross-compile and mixes the host signal definitions with those of the target. This manifests itself differently depending on the combination of host architecture and target architecture. To resolve the issue, simply re-compile bash natively. "But bash uses epatch!" you exclaim. In that case, you will need to modify the ebuild and comment out all the calls to epatch. Once you've installed the fixed bash this way, uncomment all of the bash lines and rebuild it again.

uClibc build segfaults/crashes while building locale

The uClibc locale support is pretty experimental at this point. Unless you really need support for it (and you're willing to help bang on the problem), simply disable support by adding -nls -iconv -pregen -userlocales values to the USE flags when building uClibc.


Retrieved from "https://wiki.gentoo.org/index.php?title=Embedded_Handbook/General/Full&oldid=433310"