User:Abkinch/Pi3b Cross Compile

Introduction

This article describes the process of building Gentoo for the Raspberry Pi from scratch in 64-bit. It differs from the existing Raspberry Pi 64-bit documentation (REF) in that it does not assume an existing Stage3 tarball.

The Raspberry Pi 3 Model B differs from previous models in that it's ARMv8 Broadcom BCM2837 chip is 64-bit and has 4 cores. It integrated wifi, bluetooth, ethernet and HDMI, all of which should be functional by the end of this process.

The process of building gentoo for the Pi 3B is likely to change rapidly, and by the time anyone is reading this article, particulars will probably be incorrect. With that in mind, the article will try to illustrate not only particular commands, but also methodology that may be applicable even with changes (within reason).

It is the hope of the author (abkinch) that this article will not only help others achieve gentoo success with the Pi3B, but that some of the patterns illustrated here may prove useful for projects on other architectures.

The author (abkinch) can often be found lurking in #gentoo-arm on the freenode IRC network (where many other active gentooers lurk as well), and is happy to field questions, corrections and suggestions.

Preparing the host system

Definitions

Here are a couple of definitions to get started:

• host system (or host) - the computer used to perform the initial build and configuration. This system is assumed to be running a modern Gentoo install and be x86_64 (though theoretically, x86_64 is not mandatory). It's probably best to use a very fast system for this; it'll be doing a lot of compiling.
• target system (or target) - the Raspberry Pi 3B that we intend to run Gentoo on.
• cross-compiler - a compiler toolchain (binutils, glibc, kernel headers and gcc) capable of running on one architecture (x86_64) and building for another (arm64).
• qemu - a tool that lets us run programs on one architecture (x86_64) that are intended for another (arm64). Critical for this build process.

Configuring the host kernel

The host system needs to have a few features enabled in the kernel in order to perform all of the functions we will need. Any tristate (y/n/m) options can be configured as modules without affecting the process.

Reconfigure the host system kernel with the following options enabled:

• CONFIG_BINFMT_MISC - this allows qemu to execute aarch64 (arm64) binaries without being explicitly called, necessary for the chroot environment that will be used later.
• CONFIG_KVM and ( CONFIG_KVM_INTEL or CONFIG_KVM_AMD ) - qemu uses KVM.

... [I THINK I'M FORGETTING ONE] ...

Build, install & reboot in the usual ways.

Installing the cross-compiler

There is a handy tool, crossdev, that takes a lot of the work out of building a cross-compiler. Start by emerging crossdev:

root #emerge -av sys-devel/crossdev

Note
Almost all commands we will be running need to be run as root, so unless otherwise specified, assume root is needed.

Take a look at the options by running

root #crossdev -h

and the target options by running

root #crossdev -t help

.

We will be building for the architecture aarch64-unknown-linux-gnu.

Note
It can be easy to confuse aarch64 and arm64. They both refer to the same thing. In general, anything dealing with compiling and system architecture is aarch64; anything dealing with Gentoo and emerge is arm64 (e.g. use flags)

One way to go here is just use build the toolchain with the latest stable versions:

root #crossdev --stable -t aarch64-unknown-linux-gnu

However, at the time of writing, the latest "stable" gcc is 5.4.x, but significant improvements have been made for arm64 in newer {{c|gcc}. So, we will use the following versions:

• gcc - 7.2.0
• glibc - 2.26
• binutils - 2.28
• linux-headers - 4.10

To build the cross-compiler toolchain, run:

root #crossdev -v --gcc 7.2.0 --glibc 2.26 --binutils 2.28 --linux 4.10 -t aarch64-unknown-linux-gnu

Get a cup of coffee. This build will take a little while.

After the compile, the directory /usr/aarch64-unknown-linux-gnu should contain your toolchain, and you'll have access to utilities like aarch64-unknown-linux-gcc and aarch64-unknown-linux-emerge which will be used extensively below.

Note
As a side none, aarch64-unknown-linux-emerge is really just a wrapper script for the host install of emerge, but it sets up some environment variables that will make our work easier. This could all be done by just using emerge with the right environment settings, and occasionally when debugging the process it might help to do this. Take a look inside the wrapper script to see how it does its magic.

(optional) Verifying the cross-compiler

It's not a bad idea to test that the compiler produces expected results. This can be done by compiling a small program with the cross compiler and verifying it produces the correct type of file. Create a file called test.c with the following contents:

FILE test.cA simple test C program

#include <stdio.h>

int main(int argc, char **argv) {
   printf("Goodbye, cruel world!\n");
   return 0;
}

Then run:

user $aarch64-unknown-linux-gnu -o test test.c

This should produce a file named test. Now check the file type:

user $file test

test: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, stripped, with debug_info

The important things here are that it says ELF 64-bit and ARM aarch64. If this isn't the case, try building crossdev again. Otherwise, it looks like the cross-compiler is producing the right kinds of files. The host system doesn't have the tools yet to further test that the program produces the expected results.

Cross-compiler portage config

Now that the cross-compiler is built, we need to setup how it will build things. There is a version of /etc/portage in /usr/aarch64-unknown-linux-gnu/etc/portage for this purpose. The options are the same as /etc/portage, but some values need to be set that one does not usually bother with on host systems.

First, set the values in the /usr/aarch64-unknown-linux-gnu/etc/portage/make.conf:

FILE /usr/aarch64-unknown-linux-gnu/etc/portage/make.conf

CHOST=aarch64-unknown-linux-gnu
CBUILD=x86_64-pc-linux-gnu
ARCH=arm64
HOSTCC=x86_64-pc-linux-gnu-gcc
ROOT=/usr/${CHOST}/

ACCEPT_KEYWORDS="arm64 ~arm64"
USE="${ARCH} -pam -acl -seccomp"

CFLAGS="-O2 -pipe -march=armv8-a+crc -mtune=cortex-a53 -ftree-vectorize -fomit-frame-pointer"
CXXFLAGS="${CFLAGS}"

FEATURES="-collision-protect buildpkg"
PKGDIR=/usr/${CHOST}/packages/
PORTAGE_TMPDIR=/usr/${CHOST}/tmp/

ELIBC="glibc"

PKG_CONFIG_PATH="${ROOT}usr/lib/pkgconfig/"

MAKEOPTS="-j13"

Change the MAKEOPTS line to reflect the number of parallel processes you want in a build (the standard formula is <number of cores> + 1).

Pay special attention to ROOT, CHOST, CBUILD. These tell portage how to build packages and where to install them.

Important
Make sure the FEATURES contains buildpkg. This process depends on building binary packages, and forgetting to set this will waste a lot of time and effort.

Note, also, the CFLAGS line. These are the recommended compiler options for the Pi 3b.

Next, set the crossdev environment needs to know what profile to use. We want this one for now (can be changed later): /usr/portage/profiles/default/linux/arm64/13.0

To set the profile, run:

root #cd /usr/aarch64-unknown-linux-gnu/etc/portage

root #rm make.profile

root #ln -s /usr/portage/profiles/default/linux/arm64/13.0 make.profile

Finally, we need to set some accept_keywords in order to have our crossdev build the correct versions of binutils/glibc/gcc. Create a directory named package.accept_keywords under /usr/aarch64-unknown-linux-gnu/etc/portage and create a file called toolchain (name is arbitrary) in it, containing:

FILE /usr/aarch64-unknown-linux-gnu/etc/portage/package.accept_keywords/toolchain

=sys-devel/gcc-7.2.0 **
=sys-devel/binutils-2.28.1 **
=sys-libs/binutils-libs-2.28 **
=sys-libs/glibc-2.26 **

Be sure to fix these lines with whatever versions you chose when running crossdev.

Portage is now configured to properly build packages for the crossdev environment.

Setting up qemu

Note
This step is a little out of order, as we won't use qemu until a while later in the process. However, it's nice to have on hand to test things throughout, and it's a necessary step to getting the host system configured. If you would prefer, you can skip this step and come back to it later when it's time to do the qemu chroot.

qemu will let us run cross-compiled aarch64 programs natively on x86_64. This will be used as an intermediate stage in the build process, and also is quite useful for testing along the way. There are two steps to getting qemu going: 1) installing, 2) setting up binfmt (the interface to BIN_MISC to execute aarch64 binaries.

Installing qemu

Important
Everything in the rest of this section assumes that your host kernel has been compiled with the parameters discussed above. If you haven't yet, go back and make sure your kernel has the necessary parameters.

This part is straight forward. Run:

root #QEMU_SOFTMMU_TARGETS=aarch64 QEMU_USER_TARGETS=aarch64 USE="static-user static-libs" emerge -av qemu

Once this completes, the binary /usr/bin/qemu-aarch64 should exist. You can try running on the test program created above with

root #qemu-aarch64 test

/lib/ld-linux-aarch64.so.1: No such file or directory

. The program fails because it doesn't know how to find glibc, but it wouldn't get this far if qemu wasn't working.

That's all there is to installing qemu for the host environment.

Setting up binfmt

Earlier, while preparing the kernel we enabled CONFIG_BIN_MISC. This feature allows us to tell the kernel that we want files with certain beginning signatures to be executable, and to execute them under a specified program. There are three ways to do this, depending on your needs:

Option 1: temporarily activate by hand

This option isn't recommended as it will not survive a host system reboot, but it's good to know how to do it. To temporarily tell the kernel how to run aarch64 programs, execute the following:

root #echo ':qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64:OC' > /proc/sys/fs/binfmt_misc/register'

Important
There should be no line breaks in that command, even though your browser may render it that way!

This will create a file the file /proc/sys/fs/binfmt_misc/qemu-aarch64. It is safe to cat this file, which will give you useful information on what has been configured.

If at any point you wish to disable binfmt_misc, run:

root #echo -1 > /proc/sys/fs/binfmt_misc/qemu-aarch64

. The kernel will unregister the configuration and the file qemu-aarch64 in that directory will disappear.

Option 2: persistently enable using OpenRC

Whether you use Option 2 or Option 3 depends on whether you are using OpenRC or systemd on the host system. If you are using OpenRC (the default for gentoo installs), start and enable the qemu-binfmt service:

root #rc-update add qemu-binfmt default

root #service qemu-binfmt start

After starting the services, verify that /proc/sys/fs/binfmt_misc/qemu-aarch64 exists.

Option 3: persistently enable using systemd

If the host system uses systemd, systemd has a service called systemd-binfmt.servce to configure binary formats.

Create a file the file /etc/binfmt.d/qemu-aarch64.conf containing:

FILE /etc/binfmt.d/qemu-aarch64.conf

# AArch64 binaries.
:qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64:OC

Now, reload daemons and restart the service (it should already be enabled unless it's been intensionally disabled):

root #systemctl daemon-reload

root #systemctl restart systemd-binfmt.service

Again, verify that /proc/sys/fs/binfmt_misc/qemu-aarch64 exists.

At this point, the host system is ready to start cross-compiling gentoo for the Raspberry Pi 3b.

Building (most of) @system

A note on the method

Any base Gentoo system is expected to have at least the full set of ebuilds in @system included. The main focus of what follows is building @system for the Pi. There is no particular reason other packages can't be built in the cross-compilation stage, and there is good reason to do so. Cross-compilation (depending on the host system) is likely to be much faster than building either in qemu or on the target system (Pi). However, some packages fail to cross-compile correctly.

There are multiple ways out there to go about cross-compiling {{c|@system} [REFS HERE]. This method has worked well for the author with a minimum of hacking and breakage.

The method proceeds in the following steps:

1. build all @system packages that don't fail under crossdev, making binary packages (for all steps)
2. install binary packages in a directory that will be used for qemu chroot
   1. build broken packages under qemu chroot
   2. build further dependencies under crossdev until they break
   3. go back to qemu chroot for broken packages, repeat until @system is complete
3. use binary packages to install on the Pi microSD card
4. qemu chroot into the microSD card and re-install the binary packages (reasons explained later)
5. configure, etc...

Depending on the user's requirements, some steps could be skipped.

Initial @system build with crossdev

Getting the process started for building @system is straight forward, but takes a while to complete. Start the build using:

root #aarch64-unknown-linux-gnu --exclude=sys-devel/gcc --exclude=dev-lang/perl -avu --keep-going @system

This will build a lot of packages and install them under /usr/aarch64-unknown-linux-gnu. The --keep-going option tells emerge to just keep building packages even if one fails along the way (skipping other packages that depend on it). This should build the majority of @system packages successfully.

Note
It's a good idea to keep track of which packages have failed as you work along in this process. You can see which packages have failed when emerge completes.

Warning
gcc is intentionally excluded here. Do not include gcc in this build. It will be dealt with in the next step.

It probably isn't necessary to exclude perl at this point. Perl (and subsequent perl modules) will not cross-compile correctly. It will build, but not be functional. It's excluded here just-in-case, and as a reminder. Perl isn't actually part of the @system build currently.

If all went well, many of the core packages are now built and have binary packages in /usr/aarch64-unknown-linux-gnu/packages (it wouldn't hurt to verify that they are there).

Cross-compiling gcc

gcc was excluded in the previous step for two reasons:

1. As of the writing of this article, gcc will fail to build as long as the -march and -mtune parameters are present in the FLAGS.
2. More importantly, we are building into the same directory that our cross-compiler lives in, and building gcc on top of it may overwrite it, and we will want the crossdev compiler functional throughout.

Note
The fact that this is also our crossdev toolchain directory is the one reason this directory is not used as the source for the install later.

Before compiling gcc, there may be additional dependencies that have not been installed yet. Since gcc was excluded, any dependencies needed only by gcc will not be build. gcc dependencies can be build with the following:

root #aarch64-unknown-linux-gnu-emerge -av --onlydeps gcc

We need to do two things to compile gcc. First, we need to disable -march and -mtune. Second, we need to be sure gcc builds as a binary package only, and does not install.

To accomplish this, run:

root #CFLAGS="-O2 -pipe -ftree-vectorize -fomit-frame-pointer" CXXFLAGS=${CFLAGS} aarch64-unknown-linux-gnu-emerge -av --buildpkg --buildpkgonly sys-devel/gcc"

This will build a functional gcc package for aarch64 without installing it. It won't be fully optimized, but as an optional later step, an optimized gcc can be build either under qemu or on the native Pi install once it's complete.

The @system build should be complete enough at this point that it can be run under chroot.

Creating our "build" directory

We will now create our build directory. This will mimic the final system to help build the remaining binary packages that failed.

The build directory will be assumed to be at /usr/aarch64-unknown-linux-gnu/build/root. Create this directory.

To install the current binary packages in the build directory, run:

root #ROOT=/usr/aarch64-unknown-linux-gnu/build/root aarch64-unknown-linux-gnu-emerge --usepkg --usepkgonly -avu --keep-going @system

There should now be a mostly complete install in /usr/aarch64-unknown-linux-gnu/build/root.

Configuring the chroot environment

[TODO]

Entering the qemu chroot

Warning
This step requires a fully functional qemu setup. If you haven't done it yet, make sure to complete the section #Setting up qemu.

We will setup the qemu chroot exactly like any other chroot. The directory and its programs are already in place, and the kernel knows how to execute aarch64 binaries. Once the chroot is accomplished, the binaries will also be able to find their libraries. We will be entering and leaving this (and other) chroot(s) multiple times, so we'll create two scripts.

FILE chroot-aarch64

#!/bin/bash

TARGET=aarch64-unknown-linux-gnu
ARCH_DIR=/usr/${TARGET}
KERNEL_SOURCES=/usr/src/linux-4.10.9999-raspberrypi
BUILD_DIR=${ARCH_DIR}
RUN_SHEll="/bin/bash --login"
CHROOT=/usr/bin/chroot

if [ ! -z ${1+x} ]; then
	BUILD_DIR=$1;
fi

if [ ! -z ${2+x} ]; then
	RUN_SHELL=$2;
fi

echo Chrooting to: $BUILD_DIR
cd $BUILD_DIR

mkdir -p proc dev lib/modules sys usr/portage usr/src/linux usr/${TARGET}/packages
mount -t proc none proc
mount -o bind /dev dev
mount -o bind /usr/portage usr/portage
mount -o bind $KERNEL_SOURCES usr/src/linux
mount -o bind /lib/modules lib/modules
mount -o bind /sys sys
mount -o bind ${ARCH_DIR}/packages usr/${TARGET}/packages
cp /etc/resolv.conf etc/resolv.conf
mount -o bind /dev/pts dev/pts #only for X
 
$CHROOT . $RUN_SHELL
 
umount dev/pts
umount usr/${TARGET}/packages
umount sys
umount lib/modules
umount usr/src/linux
umount usr/portage
umount dev
umount proc

This script can be anywhere in the system and set executable. /usr/local/bin is a reasonable spot, so that it's in the execution path. Note that, among other necessary mounts, this script will make /usr/portage and /usr/aarch64-unknown-linux-gnu/packages available within the chroot, appearing to be in the same spot.

FILE initchroot.sh

#!/bin/bash

locale-gen
gcc-config -l
ldconfig -v
ROOT=/ env-update
source /etc/profile
PS1="(chroot) $PS1"

This script needs to be placed somewhere under /usr/aarch64-unknown-linux-gnu/build/root.

Finally, the chroot environment needs its own copy of /usr/bin/qemu-aarch64, so copy it there:

root #cp /usr/bin/qemu-aarch64 /usr/aarch64-unknown-linux-gnu/build/root/usr/bin/

Everything should be ready now. To enter the chroot environment, run:

root #chroot-aarch64 /usr/aarch64-unknown-linux-gnu/build/root

You should find yourself in a new shell under the chroot. Basic utilities should be available and working.

Now, initialize the environment with:

root #source initchroot.sh

Now emerge should be working within the qemu chroot environment.