Embedded Handbook/General/Full

From Gentoo Wiki
Jump to:navigation Jump to:search
This article has been flagged for not conforming to the wiki guidelines. Please help Gentoo out by starting fixing things.
The information in this article is probably outdated. You can help the Gentoo community by verifying and updating this article.


Introduction

Cross development has traditionally been a black art, requiring a lot of research, trial and error, and perseverance. Intrepid developers face a shortage of documentation and the lack of mature, comprehensive open source toolkits for multi-platform cross development. Ongoing work by the Embedded or Toolchain projects, and other contributors is yielding a Gentoo-based development platform that greatly simplifies cross development.

The toolchain

The term "toolchain" refers to the collection of packages used to build up a system (the "tools" which are used in the "chain" of events to take some input and produce some output). It is a loose definition in terms of what packages exactly are considered part of the toolchain, but for the sake of keeping things simple, we will consider the components that are needed to compile code into something fun and usable.

Your typical toolchain is therefore composed of the following:

sys-devel/binutils
Essential utilities for handling binaries (includes assembler and linker).
sys-devel/gcc
The GNU Compiler Collection (the C and C++ compiler).
sys-libs/glibc, sys-libs/uclibc-ng, or sys-libs/newlib
The system C library.
sys-kernel/linux-headers
Kernel headers needed by the system C library.
sys-devel/gdb
The GNU debugger.

All proper Gentoo systems have a toolchain installed as part of the base system. This toolchain is configured to build binaries native to its host platform.

In order to build binaries on the host system for a non-native platform you'll need a special toolchain - a so-called cross toolchain - which can target that particular platform. Gentoo provides a simple but powerful tool called crossdev for this purpose. Crossdev can build and install arbitrary GCC-supported cross toolchains on the host system, and because Gentoo installs toolchain files into target-specific directories the toolchains built by crossdev will not interfere with the host's native toolchain.

Toolchain tuples

All toolchains have a prefix (think CHOST). More details on that can be found in the system tuples article.

Environment variables

Certain environment variables used by the Gentoo toolchain and Portage can thoroughly confuse developers inexperienced with cross development. The following table explains some tricky variables and provides sample values based on the cross development examples presented in this guide. See More terminology and variables (below) for more unusual variables and related concepts.

Variable name Meaning when building cross-toolchain Meaning when building cross-binaries
CBUILD Platform you are building on Platform you are building on
CHOST Platform the cross-toolchain will run on Platform the binaries built by cross-toolchain will run on
CTARGET Platform the binaries built by cross-toolchain will run on Platform the binaries built by cross-toolchain will run on. Redundant, but there's no harm in setting this, and a few binaries do like it.
ROOT Path to the virtual root (/) you are installing into
PORTAGE_CONFIGROOT Path to the virtual root (/) Portage can find its config files (like /etc/make.conf)

Say we have an AMD64 desktop as our normal Gentoo machine and we have an ARM PDA we wanted to develop for, the above table would look like:

Variable name Value for building cross-toolchain Value for building cross-binaries
CBUILD x86_64-pc-linux-gnu x86_64-pc-linux-gnu
CHOST x86_64-pc-linux-gnu arm-unknown-linux-gnu
CTARGET arm-unknown-linux-gnu Not set.
ROOT Not set - defaults to / /path/where/you/install
PORTAGE_CONFIGROOT Not set - defaults to / /path/where/your/portage/env/for/arm/pda/is

More terminology and variables

canadian cross
The process of building a cross-compiler which will run on a different machine from the one it was compiled on (CBUILD != CHOST && CHOST != CTARGET)
sysroot
The system root is where all the cross-compiler libraries and headers are installed. In other words, every library and header the cross-compiler needs to generate cross-compiled binaries are put into this directory. In theory, this means the toolchain is good enough. In practice, the cross-compiler often wants some of a packages's library dependencies, such as ncurses, installed into sysroot first.
hardfloat
The system has a hardware Floating Point Unit (FPU) to handle floating point math
softfloat
The system lacks a hardware FPU so all floating point operations are approximated with fixed point math
PIE
Position Independent Executable (-fPIE -pie)
PIC
Position Independent Code (-fPIC)
CRT
C run time


Compiling with QEMU user chroot.


This wiki page explains the required steps needed to be able to chroot into a root filesystem whose platform (e.g. aarch64) is different from the running system (e.g. amd64). Software can then be compiled within the chroot transparently using QEMU emulation.

Installation

Kernel

The build system's kernel must support miscellaneous binary formats. This can be enabled with CONFIG_BINFMT_MISC=m or CONFIG_BINFMT_MISC=y in the the kernel's .config file.

Tip
A system restart is required after building this module before it can be used.
KERNEL Enable CONFIG_BINFMT_MISC
Executable file formats  --->
  <*> Kernel support for MISC binaries

USE Flags

USE flags for app-emulation/qemu QEMU + Kernel-based Virtual Machine userland tools

accessibility Adds support for braille displays using brltty
aio Enables support for Linux's Async IO
alsa Enable alsa output for sound emulation
bpf Enable eBPF support for RSS implementation.
bzip2 Enable bzip2 compression support
capstone Enable disassembly support with dev-libs/capstone
curl Support ISOs / -cdrom directives via HTTP or HTTPS.
debug Enable extra debug codepaths, like asserts and extra output. If you want to get meaningful backtraces see https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Backtraces
doc Add extra documentation (API, Javadoc, etc). It is recommended to enable per package instead of globally
fdt Enables firmware device tree support
filecaps Use Linux file capabilities to control privilege rather than set*id (this is orthogonal to USE=caps which uses capabilities at runtime e.g. libcap)
fuse Enables FUSE block device export
glusterfs Enables GlusterFS cluster fileystem via sys-cluster/glusterfs
gnutls Enable TLS support for the VNC console server. For 1.4 and newer this also enables WebSocket support. For 2.0 through 2.3 also enables disk quorum support.
gtk Add support for x11-libs/gtk+ (The GIMP Toolkit)
infiniband Enable Infiniband RDMA transport support
io-uring Enable efficient I/O via sys-libs/liburing.
iscsi Enable direct iSCSI support via net-libs/libiscsi instead of indirectly via the Linux block layer that sys-block/open-iscsi does.
jack Add support for the JACK Audio Connection Kit
jemalloc Enable jemalloc allocator support
jpeg Enable jpeg image support for the VNC console server
keyutils Support Linux keyrings via sys-apps/keyutils
lzo Enable support for lzo compression
multipath Enable multipath persistent reservation passthrough via sys-fs/multipath-tools.
ncurses Enable the ncurses-based console
nfs Enable NFS support
nls Add Native Language Support (using gettext - GNU locale utilities)
numa Enable NUMA support
opengl Add support for OpenGL (3D graphics)
oss Add support for OSS (Open Sound System)
pam Add support for PAM (Pluggable Authentication Modules) - DANGEROUS to arbitrarily flip
pin-upstream-blobs Pin the versions of BIOS firmware to the version included in the upstream release. This is needed to sanely support migration/suspend/resume/snapshotting/etc... of instances. When the blobs are different, random corruption/bugs/crashes/etc... may be observed.
pipewire Enable pipewire output for sound emulation
plugins Enable qemu plugin API via shared library loading.
png Enable png image support for the VNC console server
pulseaudio Enable pulseaudio output for sound emulation
python Add optional support/bindings for the Python language
rbd Enable rados block device backend support, see https://docs.ceph.com/en/mimic/rbd/qemu-rbd/
sasl Add support for the Simple Authentication and Security Layer
sdl Enable the SDL-based console
sdl-image SDL Image support for icons
seccomp Enable seccomp (secure computing mode) to perform system call filtering at runtime to increase security of programs
selinux !!internal use only!! Security Enhanced Linux support, this must be set by the selinux profile or breakage will occur
slirp Enable TCP/IP in hypervisor via net-libs/libslirp
smartcard Enable smartcard support
snappy Enable support for Snappy compression (as implemented in app-arch/snappy)
spice Enable Spice protocol support via app-emulation/spice
ssh Enable SSH based block device support via net-libs/libssh2
static Build the User and Software MMU (system) targets as well as tools as static binaries
static-user Build the User targets as static binaries
systemtap Enable SystemTAP/DTrace tracing
test Enable dependencies and/or preparations necessary to run tests (usually controlled by FEATURES=test but can be toggled independently)
udev Enable virtual/udev integration (device discovery, power and storage device support, etc)
usb Enable USB passthrough via dev-libs/libusb
usbredir Use sys-apps/usbredir to redirect USB devices to another machine over TCP
vde Enable VDE-based networking
vhost-net Enable accelerated networking using vhost-net, see https://www.linux-kvm.org/page/VhostNet
virgl Enable experimental Virgil 3d (virtual software GPU)
virtfs Enable VirtFS via virtio-9p-pci / fsdev. See https://wiki.qemu.org/Documentation/9psetup
vnc Enable VNC (remote desktop viewer) support
vte Enable terminal support (x11-libs/vte) in the GTK+ interface
xattr Add support for getting and setting POSIX extended attributes, through sys-apps/attr. Requisite for the virtfs backend.
xen Enables support for Xen backends
zstd Enable support for ZSTD compression
Data provided by the Gentoo Package Database · Last update: 2024-03-23 08:35 More information about USE flags

Important
app-emulation/qemu must be emerged with USE=static-user to use user targets.
Warning
QEMU_SOFTMMU_TARGETS and QEMU_USER_TARGETS are empty by default and must be defined to utilize user targets.
Note
Activating the static-user will require supporting libraries to be build with static-libs support.
FILE /etc/portage/package.use/qemuEnable aarch64 user target and static-libs in supporting libraries.
# Enable static-user and add the aarch64 target
app-emulation/qemu static-user QEMU_SOFTMMU_TARGETS: aarch64 QEMU_USER_TARGETS: aarch64
# required by app-emulation/qemu::gentoo[static,static-user]
# required by qemu (argument)
dev-libs/glib static-libs
# required by app-emulation/qemu::gentoo[-static,static-user]
# required by qemu (argument)
sys-libs/zlib static-libs
# required by app-emulation/qemu::gentoo[-static,static-user,xattr]
# required by qemu (argument)
sys-apps/attr static-libs
# required by dev-libs/glib::gentoo
# required by app-emulation/qemu::gentoo[-static,static-user]
# required by qemu (argument)
dev-libs/libpcre2 static-libs
Warning
If using LTO and the static-user, GCC will use huge amount of RAM when compiling. Because of this, it is generally recommended to disable LTO while compiling in this configuration or use Clang if LTO is required.
See also
bug #883419
Note
When xattr is not enabled emerge may fail to install packages in the chroot'ed environment.

QEMU target configuration

By default, app-emulation/qemu does not define any QEMU_SOFTMMU_TARGETS or QEMU_USER_TARGETS. The example configuration above only includes aarch64 targets.

Tip
The output of emerge -pv app-emulation/qemu contains the full target list.
Tip
To build for a Raspberry Pi running a 32-bit stage3 build, be sure to select QEMU_USER_TARGETS="arm" here. For 64-bit stage3 builds, select aarch64. When in doubt both can be set.

To build all targets:

root #echo 'QEMU_SOFTMMU_TARGETS="*"' >> /etc/portage/make.conf
root #echo 'QEMU_USER_TARGETS="*"' >> /etc/portage/make.conf

Alternatively, the USE_EXPAND syntax can be used to keep make.conf free of USE_EXPAND values:

FILE /etc/portage/package.use/qemuConfigure QEMU to build all targets.
app-emulation/qemu QEMU_SOFTMMU_TARGETS: * QEMU_USER_TARGETS: *

Emerge

root #emerge --ask --update --newuse --deep app-emulation/qemu

Configuration

Group configuration

For non-root users to use QEMU, they must be added to the kvm group.

To add larry to the kvm group:

root #gpasswd -a larry kvm

OpenRC

To start the qemu-binfmt service:

root #rc-service qemu-binfmt start

It may be wise for the services to be started by default on boot:

root #rc-update add qemu-binfmt default

systemd

The systemd-binfmt service must be configured by adding files containing the desired handler registration strings to /etc/binfmt.d/.

Modern versions of qemu ship a binfmt configuration file that supports all binary formats. Simply link it to /etc for binary format support, then skip the following manual file creation steps.

root #ln -s /usr/share/qemu/binfmt.d/qemu.conf /etc/binfmt.d/qemu.conf

Manual binfmt configuration

Alternatively, to only make support for aarch64 and arm architectures, separate files can manually be created:

FILE /etc/binfmt.d/qemu-aarch64-static.confSpecifying an aarch64 binfmt-misc handler for systemd-binfmt
:aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\xfc\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64:
FILE /etc/binfmt.d/qemu-arm-static.confSpecifying an arm binfmt-misc handler for systemd-binfmt
:arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm:

See #Binary format handlers for a list of possible handlers.

Note
In the FileBox above, the eighth byte of the mask has been set to fc (rather than ff), to allow programs like gcc to be executed. See for example Debian bug #799120.

With the file(s) created in /etc/binfmt.d/, systemd will now find them automatically at system boot time. See man 5 binfmt.d for more information.

Since the files were likely created just a few moments ago, it will be necessary to restart the service once to register the binary formats:

root #systemctl restart systemd-binfmt

To confirm the service is running properly after restarting:

root #systemctl status systemd-binfmt

Usage

Preparing the chroot

To be able to chroot into a system of a different platform (e.g. aarch64 while using an amd64 system), mounted in e.g. /mnt/gentoo, the QEMU static-user binary must be copied into the environment.

This file is named qemu-<architecture> under /usr/bin/ and should be copied to usr/bin/ within the build environment.

To use an aarch64 chroot environment at /mnt/gentoo-aarch64:

user $cp /usr/bin/qemu-aarch64 /mnt/gentoo-aarch64/usr/bin

Chrooting

Once the environment has been prepared, sys-apps/arch-chroot can be used:

root #arch-chroot /mnt/gentoo-aarch64

Portage configuration

Currently QEMU does not support the pid-sandbox (bug #703278) and network-sandbox (bug #703276) Portage features.

To disable these features::

FILE /etc/portage/make.conf
FEATURES="-pid-sandbox -network-sandbox"

Additional Usage

Binary format handlers

Binary format handlers look rather confusing at first, but they can be simply broken down as: :name:type:offset:magic:mask:interpreter:flags[1]

With each field representing, in order of appearance:

  • name being the name of the binary format, a new /proc file will be created with this name at /proc/sys/fs/binfmt_misc
  • type being either E or M
    • E means the executable file format is identified by its file extension
    • M means the format is identified by the magic number mask
  • offset represents the offset of the magic/mask in the file in bytes, the default is 0
  • magic is a byte sequence that binfmt matches for to know what files to pass through to the interpreter
  • interpreter is a program that is to be run with the matching file as an argument
  • flags is a field that controls different aspects of interpreter when it is invoked
    • P - preserve-argv[0]: adds an argument to the argument vector to preserve the original argv[0]
    • O - open-binary: opens the file for reading and pass its descriptor as an argument rather than the full path
    • C - credentials: current behavior of binfmt_misc is to calculate the credentials and security of the new process according to the interpreter, but including this flag will be calculated according to the binary. (Implies the O flag)
    • F - fix binary: currently, the binary is spawned lazily when the misc format file is invoked, however, this does not work very well in the face of mount namespaces and changroots so F allows the binary to be opened always once it is installed, regardless of environment changes.

Some restrictions do apply with binfmt:

  • The whole register string may not exceed 1920 characters
  • The magic must reside in the first 128 bytes of the file
  • The interpreter string may not exceed 127 characters
Architecture
 Binfmt Handler String
AARCH64 :aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\xfc\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64:
ARM :arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm:
ARMeb :armeb:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-armeb:
Alpha :alpha:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x26\x90:\xff\xff\xff\xff\xff\xfe\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-alpha:
DOS :DOSWin:M::MZ::/etc/eselect/wine/bin/wine:
LOONGARCH64 :loongarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x02\x01:\xff\xff\xff\xff\xff\xff\xff\xfc\x00\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-loongarch64:
M68K :m68k:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x04:\xff\xff\xff\xff\xff\xff\xfe\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-m68k:
MIPS :mips:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-mips:
MIPS64 :mips64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xf3\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-mips64:
MIPSel :mipsel:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-mipsel:
MIPS N32 :mipsn32:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-mipsn32:
MIPSel N32 :mipseln32:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-mipseln32:
PPC :ppc:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x14:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-ppc:
PPC64 :ppc64:M::\x7fELF\x02\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x15:\xff\xff\xff\xff\xff\xff\xff\xfc\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-ppc64:
RISCV64 :riscv64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xf3\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-riscv64:
SH4 :sh4:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x2a\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfb\xff\xff\xff:/usr/bin/qemu-sh4:
SH4eb :sh4eb:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x2a:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-sh4eb:
SPARC :sparc:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x02:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-sparc:

Manual setup

First, mount the binfmt_misc handler if it is not already mounted, then register the format with the kernel via the procfs:

root #[ -d /proc/sys/fs/binfmt_misc ] || modprobe binfmt_misc
root #[ -f /proc/sys/fs/binfmt_misc/register ] || mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc
Important
Do not register a handler that matches the host machine (in this example, registers for x86 and AMD64 has been excluded since the host machine).

To register a handler, pipe its format to /proc/sys/fs/binfmt_misc/register, for example

root #echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm:' > /proc/sys/fs/binfmt_misc/register
See also
#Binary format handlers for a list of possible handlers.

Advanced Tips

Sometimes we'll need to pass additional args to QEMU (CPU model), so we'll create a wrapper script (in C) that'll call QEMU with it:

FILE qemu-wrapper.c
/*
 * Pass arguments to qemu binary
 */

#include <string.h>
#include <unistd.h>

int main(int argc, char **argv, char **envp) {
	char *newargv[argc + 3];

	newargv[0] = argv[0];
	newargv[1] = "-cpu";
	newargv[2] = "cortex-a8"; /* here you can set the cpu you are building for */

	memcpy(&newargv[3], &argv[1], sizeof(*argv) * (argc -1));
	newargv[argc + 2] = NULL;
	return execve("/usr/bin/qemu-arm", newargv, envp);
}

Compile the wrapper with:

root #gcc -static qemu-wrapper.c -O3 -s -o qemu-wrapper

Then copy into the chroot. Notice the first example ARM entry in the binfmt_misc section uses this method.

References

  1. ↑ https://docs.kernel.org/admin-guide/binfmt-misc.html

Creating a cross-compiler

The first thing users should know about building a toolchain is that some versions of toolchain components refuse to work together. Exactly which combinations are problematic is a matter that's constantly in flux as the Gentoo ebuild repository evolves. The only reliable way to determine what works is to run crossdev, adjusting individual component versions as necessary, until crossdev completes the toolchain build successfully. Even then, the cross toolchain may build binaries which break on the target system. Only through trial, error, and patience will one arrive at a favorable combination of all factors.

Users do not have to worry about the cross-compiler interfering with the native build system. All of the toolchain packages are designed such that they are isolated from each other based on the target. This way cross-compilers can be installed for desired architecture(s) without breaking the rest of the system.

crossdev

Intro

Generating a cross-compiler by hand is a long and painful process. This is why it has been fully integrated into Gentoo! A command-line front-end called crossdev will run emerge with all of the proper environment variables and install all the right packages to generate arbitrary cross-compilers based on the need of the user.

First, install Crossdev:

root #emerge --ask sys-devel/crossdev

Consider installing the unstable version of crossdev to get all the latest fixes.

Note
Those upgrading from older versions of crossdev, and have crossdev-wrappers installed, need to depclean crossdev-wrappers first. The existing cross-toolchains will remain intact.

Only basic usage of crossdev is covered here, but crossdev can customize the process fairly well for most needs. Run crossdev --help to get some ideas on how to use crossdev. Here are some common usage options:

crossdev --g [gcc version] --l [(g)libc version] --b [binutils version] --k [kernel headers version] -P -v -t [tuple]
crossdev -S -P -v -t [tuple]

Installing

First, go set up an overlay as described on the crossdev page.

Then you must select the proper tuple for the target. Here, it will be assumed that a cross-compiler for the SH4 (SuperH) processor with glibc running on Linux is desired to be built by the user. This action will be performed on a PowerPC machine. Generate a SH4 cross-compiler:

root #crossdev --target sh4-unknown-linux-gnu
-----------------------------------------------------------------------------------------------------
 * Host Portage ARCH:     ppc
 * Target Portage ARCH:   sh
 * Target System:         sh4-unknown-linux-gnu
 * Stage:                 4 (C/C++ compiler)

 * binutils:              binutils-[latest]
 * gcc:                   gcc-[latest]
 * headers:               linux-headers-[latest]
 * libc:                  glibc-[latest]

 * PORTDIR_OVERLAY:       /var/db/repos/local
 * PORT_LOGDIR:           /var/log/portage
 * PKGDIR:                /usr/portage/packages/powerpc-unknown-linux-gnu/cross/sh4-unknown-linux-gnu
 * PORTAGE_TMPDIR:        /var/tmp/cross/sh4-unknown-linux-gnu
  _  -  ~  -  _  -  ~  -  _  -  ~  -  _  -  ~  -  _  -  ~  -  _  -  ~  -  _  -  ~  -  _  -  ~  -  _  
 * Forcing the latest versions of {binutils,gcc}-config/gnuconfig ...                          [ ok ]
 * Log: /var/log/portage/cross-sh4-unknown-linux-gnu-binutils.log
 * Emerging cross-binutils ...                                                                 [ ok ]
 * Log: /var/log/portage/cross-sh4-unknown-linux-gnu-gcc-stage1.log
 * Emerging cross-gcc-stage1 ...                                                               [ ok ]
 * Log: /var/log/portage/cross-sh4-unknown-linux-gnu-linux-headers.log
 * Emerging cross-linux-headers ...                                                            [ ok ]
 * Log: /var/log/portage/cross-sh4-unknown-linux-gnu-glibc.log
 * Emerging cross-glibc ...                                                                    [ ok ]
 * Log: /var/log/portage/cross-sh4-unknown-linux-gnu-gcc-stage2.log
 * Emerging cross-gcc-stage2 ...                                                               [ ok ]
Note
At the moment it is not possible to set PORTAGE_CONFIGROOT before calling crossdev to a folder set to the architecture for the target. A custom config must be used. To use arch specific USE flags, like altivec in a non-PowerPC architecture, unmask the use flag in /var/db/repos/gentoo/profiles/base/use.mask, or temporarily change the system profile.

Quick test

If everything goes as planned, a shiny new compiler should now be present on the machine. Give it a spin!

Use the SH4 cross-compiler:

user $sh4-unknown-linux-gnu-gcc --version
sh4-unknown-linux-gnu-gcc (GCC) 4.2.0 (Gentoo 4.2.0 p1.4)
Copyright (C) 2007 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
user $echo 'int main(){return 0;}' > sh4-test.c
user $sh4-unknown-linux-gnu-gcc -Wall sh4-test.c -o sh4-test
user $file sh4-test
sh4-test: ELF 32-bit LSB executable, Renesas SH, version 1 (SYSV), for GNU/Linux 2.6.9, dynamically linked (uses shared libs), not stripped

If the crossdev command failed, the log file may be reviewed to see if the problem is local. If unable to fix the issue, a bug may be filed in Bugzilla.

Tuples

To find out which tuple should be used, look over the output from the following command:

root #crossdev -t help

There should now be a newly compiled cross-compiler in the sysroot at /usr/${CTARGET}/. It's a good idea to create pre-built binary packages so as to not end up waiting another two to three hours every time this toolchain should be reinstalled.

Create binpkgs

Note
The default value for creating binary packages is xpak. For more information on managing binary packages please refer to the Binary package guide.
root #quickpkg --include-unmodified-config=y cross-sh4-unknown-linux-gnu/gcc
root #quickpkg --include-unmodified-config=y cross-sh4-unknown-linux-gnu/glibc
root #quickpkg --include-unmodified-config=y cross-sh4-unknown-linux-gnu/binutils
root #quickpkg --include-unmodified-config=y cross-sh4-unknown-linux-gnu/linux-headers

If the quickpkg command warns about excluded files, please follow its prompts to include all files.

In the future the sysroot can be reinstalled by executing the following simple Portage command:

root #emerge -k cross-sh4-unknown-linux-gnu/gcc cross-sh4-unknown-linux-gnu/glibc cross-sh4-unknown-linux-gnu/binutils cross-sh4-unknown-linux-gnu/linux-headers
Tip
For the sake of solidarity (to prevent inferences), the definition and subsequent explanation of sysroot are located in the Introduction and outlined under Cross-compiler internals below; respectively.

Uninstalling

To uninstall a toolchain, simply use the --clean option. If the sysroot was modified by hand, there will be a prompt to delete every file inside, so it is possible to prepend yes | to this command if there is no doubt about what can be deleted:

Uninstall the SH4 cross-compiler:

root #crossdev --clean sh4-unknown-linux-gnu

Deleting any and all files in the /usr/${CTARGET}/ directory should be completely safe.

Cross-compiler internals

Tip
This section is included for posterity and in the hopes that others will find it useful. The target audience is all users who (due to curiosity, naïveté, general misconceptions, and/or their current inability to comprehend this warning) really really want to create their own cross compiler with binutils/(glibc/musl/some other libc)/gcc all by themselves. This section is a synopsys, and lacks the relevant documentation that would prevent the myriad of build failures that may be encountered. If such help is required, see the Beyond section in the handbook for some pointers. Please do not ask for any support at all for this.
Warning
If still reading, it is a very good idea to check out the crosstool project (again refer to the Beyond section) as that provides a distribution independent method for generating cross-compilers. While imperfect, it is certainly the best (and really only) option out there for creating cross-compilers.

Overview

There are generally two ways to build a cross-compiler. The "accepted" way, and the cheater's shortcut.

The current "accepted" way is:

  1. binutils
  2. kernel headers
  3. libc headers
  4. gcc stage1 (c-only)
  5. libc
  6. gcc stage2 (c/c++/etc...)

The cheater's shortcut is:

  1. binutils
  2. kernel headers
  3. gcc stage1 (c-only)
  4. libc
  5. gcc stage2 (c/c++/etc...)

The reason people are keen on the shortcut is that the libc headers step tends to take quite a while, especially on slower machines. It can also be kind of a pain to setup kernel/libc headers without a usable cross compiler. Note, though, that if help with cross-compilers is sought, upstream projects will not want to help if the shortcut was taken.

Also note that the shortcut requires the gcc stage1 to be "crippled". Since building without headers, the sysroot option cannot be enabled nor proper gcc libs can be built. This is okay if the only thing that is being used in the stage1 is building the C library and a kernel, but beyond that, a nice sysroot-based compiler is needed.

The "accepted" way is described below as the steps are pretty much the same. Some extra patches are needed for gcc in order to take the shortcut.

Sysroot

The cross-compiling will be done using the sysroot method. But what does the sysroot do?

The sysroot tells GCC to consider dir as the root of a tree that contains (a subset of) the root filesystem of the target operating system. Target system headers, libraries and run-time object files will be searched in there.

The top level directory is commonly rooted in /usr/$CTARGET

CODE A typical sysroot layout
|-- bin/
|-- lib/            critical runtime libs (libc/ldso/etc...)
`-- usr/
    |-- include/    development headers
    |   |-- linux/      like the linux kernel
    |   `-- asm/        like the arch-specific
    `-- lib/        non critical runtime libs / development libs

As can be seen, it's just like the directory setup in / but in /usr/$CTARGET. This setup is of course not an accident but designed on purpose so applications/libraries can be easily migrated out of /usr/$CTARGET and into / on the target board. If desired, /usr/$CTARGET could be used as a quick NFS root!

Note
The old style of cross-compilers was to use --prefix=/usr/$CTARGET. If those binutils/gcc versions are used that predate sysroot support, it may be needed to do just this. It will not be documented because (1) old/crusty/busted versions should not be used and (2) it's quite a huge pain compared to sysroot.

Binutils

Grab the latest binutils tarball and unpack it.

The --disable-werror option is to prevent binutils from aborting the compile due to warnings. Great feature for developers, but a pain for users. Configure and build binutils:

root #make
root #make install DESTDIR=$PWD/install-root

The reason of install into the localdir is the crap that doesn't belong can be removed. For example, a normal install will give /usr/lib/libiberty.a which doesn't belong in the host /usr/lib. So clean out stuff first:

root #rm -rf install-root/usr/{info,lib,man,share}

And install what's left:

root #cp -a install-root/* /

Kernel headers

Grab the latest Linux tarball and unpack it. There are two ways of installing the kernel headers: sanitized and unsanitized. It is clear that the former is better (but it requires a recent version of the Linux kernel), but both will be quickly covered.

Note
Clearly the $ARCH variable need to be replaced with a value that makes sense for needed platform.

Building and install the unsanitized headers:

root #yes "" | make ARCH=$ARCH oldconfig prepare
root #mkdir -p /usr/$CTARGET/usr/include
root #cp -a include/linux include/asm-generic /usr/$CTARGET/usr/include/
root #cp -a include/asm-$ARCH /usr/$CTARGET/usr/include/asm

Build and install the sanitized headers:

root #make ARCH=$ARCH headers_install INSTALL_HDR_PATH=/usr/$CTARGET/usr

System libc headers

Grab the latest glibc tarball and unpack it. Glibc is picky, so compilation have to be done in a directory separate from the source code. Build and install the glibc headers:

root #mkdir build
root #cd build
root #../configure --host=$CTARGET --prefix=/usr --with-headers=/usr/$CTARGET/usr/include --without-cvs --disable-sanity-checks
root #make -k install-headers install_root=/usr/$CTARGET

glibc can be awkward sometimes, so have to do a few things by hand:

root #mkdir -p /usr/$CTARGET/usr/include/gnu
root #touch /usr/$CTARGET/usr/include/gnu/stubs.h
root #cp bits/stdio_lim.h /usr/$CTARGET/usr/include/bits/

GCC stage 1 (C only)

At first, help gcc find the current libc headers:

root #ln -s usr/include /usr/$CTARGET/sys-include

Then grab the latest gcc tarball and unpack it:

root #mkdir build
root #cd build
root #make
root #make install DESTDIR=$PWD/install-root

Same as binutils, gcc leaves some stuff behind that doesn't wanted. Clean the gcc stage 1:

user $rm -rf install-root/usr/{info,include,lib/libiberty.a,man,share}

System libc

Remove the old glibc build directory and recreate it:

root #rm -rf build
root #mkdir build
root #cd build
root #../configure --host=$CTARGET --prefix=/usr --without-cvs
root #make
root #make install install_root=/usr/$CTARGET

GCC stage 2 (all frontends)

A full GCC can be built up now. Whichever compiler frontends are preferred can be selected; C/C++ will just be done for simplicity. Build and install the gcc stage 2:

root #./configure --target=$CTARGET --prefix=/usr --with-sysroot=/usr/$CTARGET --enable-languages=c,c++ --enable-shared --disable-checking --disable-werror
root #make
root #make install

Core runtime files

There are many random core runtime files that people wonder what they may be for. Let's explain:

Files provided by sys-libs/glibc
File Purpose
crt0.o Older style of the initial runtime code. No one generates this anymore.
crt1.o Newer style of the initial runtime code. Contains the _start symbol which sets up the env with argc/argv/libc _init/libc _fini before jumping to the libc main. glibc calls this file 'start.S'.
crti.o Defines the function prolog; _init in the .init section and _fini in the .fini section. glibc calls this 'initfini.c'.
crtn.o Defines the function epilog. glibc calls this 'initfini.c'.
Scrt1.o Used in place of crt1.o when generating PIEs.
gcrt1.o Used in place of crt1.o when generating code with profiling information. Compile with -pg. Produces output suitable for the gprof util.
Mcrt1.o Like gcrt1.o, but is used with the prof utility. glibc installs this as a dummy file as it's useless on linux systems.
Files provided by sys-devel/gcc
File Purpose
crtbegin.o GCC uses this to find the start of the constructors.
crtbeginS.o Used in place of crtbegin.o when generating shared objects/PIEs.
crtbeginT.o Used in place of crtbegin.o when generating static executables.
crtend.o GCC uses this to find the start of the destructors.
crtendS.o Used in place of crtend.o when generating shared objects/PIEs.

The general linking order:

crt1.o crti.o crtbegin.o [-L paths] [user objects] [gcc libs] [C libs] [gcc libs] crtend.o crtn.o

External resources

GCC Cross-Compiler


Cross-compiling with Portage

Variables

There are a few important variables that will be used throughout this section:

Variable name Meaning
CBUILD The platform that (cross)compiles the binaries.
CHOST The platform that runs the compiled binaries.
ROOT The virtual / binaries are installed into.
PORTAGE_CONFIGROOT The virtual / Portage can find its config files (like make.conf).

These variables can be set by hand, but that obviously gets tedious very quickly. A better idea is to stick these into a shell script to avoid typing them each time.

Filesystem setup

Cross-compiling a system generally involves two directory structures. The differences between them can be difficult to understand, but they are important concepts to cross-compiling.

The first directory structure to consider is the sysroot. Please read the Introduction chapter for the definition of sysroot.

The second directory structure is the real root. This one is much simpler to understand: it's where the bootable stage3-like installation of Gentoo is located. This installation can then be copied to the target device.

The common convention is to use the /usr/${CTARGET}/ directory as the sysroot since the include/library directories in this tree are already encoded into the gcc cross-compiler for searching. Another directory could be used and then custom -I/-L paths could be added to the CPPFLAGS/LDFLAGS, but this has historically proven to be problematic in enough corner cases to be discouraged for practical use. In the embedded handbook, it will be assumed the sysroot is being used as the development ROOT.

A much slimmer/trimmed-down setup will be needed for the runtime system. The files removed from a normal installed package are the reason why this tree is not suitable for compiling against. If binary packages are built while installing into the sysroot, then those binary packages can be used in conjunction with the INSTALL_MASK variable to trim out most things. More information can be found in the man make.conf.

Intro: crossdev's wrappers

Simple wrapper scripts will be used to setup the environment variables to point to the appropriate locations, enabling cross-compilation using emerge. PORTAGE_CONFIGROOT and ROOT both point to the SYSROOT.

root #emerge --ask sys-devel/crossdev
Warning
crossdev's wrappers automatically configure the CBUILD and CHOST variables. Do not set them by hand in the environment!
Note
Before any cross-emerge can be started, emerge-wrapper --init needs to be run. Instructions printed by emerge-wrapper should be followed before starting cross-emerge.

These tools can be used for both installing into the development root (sysroot) and into the runtime root. For the latter, the --root option should be specified. For example if an armv4tl-softfloat-linux-gnueabi toolchain has been merged via crossdev, then command can then be invoked like a normal emerge. But using the ctarget prefix:

root #armv4tl-softfloat-linux-gnueabi-emerge pkg0 pkg1 pkg2

By default these wrappers use the --root-deps=rdeps option to avoid the host dependencies from being pulled into the deptree. This can lead to incomplete deptrees. Therefore the --root-deps option can be used alone to see the full dependency graph.

By default crossdev will link to the generic embedded profile. This is done to simplify things, but the user may wish to use a more advanced targeted profile. The profile symlink can be updated in order to do that.

root #ln -s /var/db/repos/gentoo/profiles/default/linux/arm/17.0 ${SYSROOT}/etc/portage/make.profile

To change settings (such as USE flags) for the target system, edit the standard Portage config files:

root #${EDITOR} ${SYSROOT}/etc/portage/make.conf

Sometimes there are some additional, cross-compile-incompatible tests that must be overridden for configure scripts. To make this possible, crossdev exports a few variables to force the test to get the answer it should receive. This helps prevent bloat in packages which add local functions to workaround issues it assumes the target system has because it could not run the test while cross-compiling (because, of course, it's not running on the target system). From time-to-time additional variables may need to be added to these files in /usr/share/crossdev/include/site/ in order to get a package to compile. To figure out the variable that needs added, it is often as simple as grepping the configure script for the autoconf variable (which often begins with ac_) and adding it to the appropriate target file. This becomes easier with ebuild development experience.

Uninstall

If uninstalling and deleting the work is desired, the sysroot tree can be safely removed without affecting any native packages. Refer to the "Uninstalling" section in the crossdev guide.


Cross-compiling the kernel

Sources

The relevant kernel sources should be installed first. A kernel sources package can be quickly emerged from the Gentoo ebuild repository or fetch the latest sources from kernel.org. The method for actually compiling the kernel is all the same.

The kernel should be installed into the sysroot so if desired to cross-compile packages which include kernel modules, the process will be transparent. Otherwise, the actual location where the kernel is built does not matter. Some people build all their kernels in /usr/src/ for example.

Setup cross-compiling

There are two fundamental variables that the kernel uses to select the target architecture. Normally these values are guessed based on the build environment, but of course that environment here does not match the target embedded system, so they will need to be overriden. The variables in question are ARCH and CROSS_COMPILE. The default values for both are found in the top-level Makefile and the values of both may be overridden on the command line.

The ARCH variable refers to the architecture that the kernel is being built for, as recognized by the kernel itself. So while portage and other people may use "x86", the kernel uses "i386". The arch/ subdirectory should be quickly checked to determine which architecture is to be used.

Hopefully the CROSS_COMPILE variable is pretty self-explanatory. It should be set to the prefix of the toolchain (including the trailing dash "-"). So if the toolchain is invoked as say x86_64-pc-linux-gnu-gcc, the trailing gcc should be chopped off and that's what should be used: x86_64-pc-linux-gnu-.

There is an additional variable, INSTALL_MOD_PATH, which defines where the /lib directory will be created, and all the modules stored. The kernel sources do not have to be transferred to the target device, but if any modules are built, this directory will be needed.

There are really two ways in which the system can be set up. The toplevel Makefile can be modified or the relevant variables can be overridden on the command line. Both methods are acceptable, so both will be covered. One of the following options can be chosen.

FILE MakefileThe vanilla Makefile
ARCH            ?= $(SUBARCH)
CROSS_COMPILE   ?=
FILE MakefileSet the ARCH and CROSS_COMPILE default values
ARCH            ?= arm
CROSS_COMPILE   ?= arm-unknown-linux-gnu-

Overriding on the command-line (instead of in the Makefile) would look something like this:

root #make ARCH=arm CROSS_COMPILE=arm-unknown-linux-gnu-

A little helper script can be used if it is necessary to switch between different kernel trees at the same time. The script will be referred to as xkmake:

FILE xkmake
#!/bin/sh
exec make ARCH="arm" CROSS_COMPILE="arm-unknown-linux-gnu-" INSTALL_MOD_PATH="${SYSROOT}" "$@"

Now, when building a kernel or performing any other action, xkmake should be executed instead of make.

Configure and compile

At this point, configuring and compiling the kernel is the same as any other kernel, so it won't be discussed in depth here. There are numerous HOWTOs and guides available that can cover the subject in much greater detail.

root #cd "${SYSROOT}/usr/src/linux"
root #xkmake menuconfig
root #xkmake


Frequently asked questions

I get "configure: error: C compiler cannot create executables"

This is a generic error and can be caused by just about anything. The test is pretty simple: can the requested compiler create an executable? However, this relies on many things being correct: the toolchain itself being completely sane, the compiler and compiler flags being appropriate, your environment set up properly, etc... The only way to find out the real source of the problem is to open up the generated config.log file and scroll down to where this test is run and see what exactly the error message is that the toolchain is spitting out.

"epatch" always fails in newly compiled system

The bash package does not properly cross-compile and mixes the host signal definitions with those of the target. This manifests itself differently depending on the combination of host architecture and target architecture. To resolve the issue, simply re-compile bash natively. "But bash uses epatch!" you exclaim. In that case, you will need to modify the ebuild and comment out all the calls to epatch. Once you've installed the fixed bash this way, uncomment all of the bash lines and rebuild it again.

uClibc build segfaults/crashes while building locale

The uClibc locale support is pretty experimental at this point. Unless you really need support for it (and you're willing to help bang on the problem), simply disable support by adding -nls -iconv -pregen -userlocales values to the USE flags when building uClibc.


Retrieved from "https://wiki.gentoo.org/index.php?title=Embedded_Handbook/General/Full&oldid=1273508"