User:Abkinch/Pi3b Cross Compile

Introduction
This article describes the process of building Gentoo for the Raspberry Pi from scratch in 64-bit. It differs from the existing Raspberry Pi 64-bit documentation (REF) in that it does not assume an existing Stage3 tarball.

The Raspberry Pi 3 Model B differs from previous models in that it's ARMv8 Broadcom BCM2837 chip is 64-bit and has 4 cores. It integrated wifi, bluetooth, ethernet and HDMI, all of which should be functional by the end of this process.

The process of building gentoo for the Pi 3B is likely to change rapidly, and by the time anyone is reading this article, particulars will probably be incorrect. With that in mind, the article will try to illustrate not only particular commands, but also methodology that may be applicable even with changes (within reason).

It is the hope of the author (abkinch) that this article will not only help others achieve gentoo success with the Pi3B, but that some of the patterns illustrated here may prove useful for projects on other architectures.

The author (abkinch) can often be found lurking in #gentoo-arm on the freenode IRC network (where many other active gentooers lurk as well), and is happy to field questions, corrections and suggestions.

Definitions
Here are a couple of definitions to get started:
 * host system (or host) - the computer used to perform the initial build and configuration. This system is assumed to be running a modern Gentoo install and be x86_64 (though theoretically, x86_64 is not mandatory).  It's probably best to use a very fast system for this; it'll be doing a lot of compiling.
 * target system (or target) - the Raspberry Pi 3B that we intend to run Gentoo on.
 * cross-compiler - a compiler toolchain (binutils, glibc, kernel headers and gcc) capable of running on one architecture (x86_64) and building for another (arm64).
 * qemu - a tool that lets us run programs on one architecture (x86_64) that are intended for another (arm64). Critical for this build process.

Configuring the host kernel
The host system needs to have a few features enabled in the kernel in order to perform all of the functions we will need. Any tristate (y/n/m) options can be configured as modules without affecting the process.

Reconfigure the host system kernel with the following options enabled: ... [I THINK I'M FORGETTING ONE] ...
 * CONFIG_BINFMT_MISC - this allows qemu to execute aarch64 (arm64) binaries without being explicitly called, necessary for the chroot environment that will be used later.
 * CONFIG_KVM and ( CONFIG_KVM_INTEL or CONFIG_KVM_AMD ) - qemu uses KVM.

Build, install & reboot in the usual ways.

Installing the cross-compiler
There is a handy tool, crossdev, that takes a lot of the work out of building a cross-compiler. Start by emerging crossdev:

Take a look at the options by running and the target options by running.

We will be building for the architecture.

One way to go here is just use build the toolchain with the latest stable versions: However, at the time of writing, the latest "stable" is 5.4.x, but significant improvements have been made for arm64 in newer {{c|gcc}. So, we will use the following versions:


 * gcc - 7.2.0
 * glibc - 2.26
 * binutils - 2.28
 * linux-headers - 4.10

To build the cross-compiler toolchain, run:

Get a cup of coffee. This build will take a little while.

After the compile, the directory should contain your toolchain, and you'll have access to utilities like  and  which will be used extensively below.

(optional) Verifying the cross-compiler
It's not a bad idea to test that the compiler produces expected results. This can be done by compiling a small program with the cross compiler and verifying it produces the correct type of file. Create a file called with the following contents:

Then run:

This should produce a file named. Now check the file type:

The important things here are that it says and. If this isn't the case, try building crossdev again. Otherwise, it looks like the cross-compiler is producing the right kinds of files. The host system doesn't have the tools yet to further test that the program produces the expected results.

Cross-compiler portage config
Now that the cross-compiler is built, we need to setup how it will build things. There is a version of in  for this purpose. The options are the same as, but some values need to be set that one does not usually bother with on host systems.

First, set the values in the :

Change the line to reflect the number of parallel processes you want in a build (the standard formula is + 1).

Pay special attention to, ,. These tell portage how to build packages and where to install them.

Note, also, the line. These are the recommended compiler options for the Pi 3b.

Next, set the crossdev environment needs to know what profile to use. We want this one for now (can be changed later):

To set the profile, run:

Finally, we need to set some in order to have our crossdev build the correct versions of binutils/glibc/gcc. Create a directory named under  and create a file called  (name is arbitrary) in it, containing:

Be sure to fix these lines with whatever versions you chose when running.

Portage is now configured to properly build packages for the crossdev environment.

Setting up
will let us run cross-compiled programs natively on. This will be used as an intermediate stage in the build process, and also is quite useful for testing along the way. There are two steps to getting going: 1) installing, 2) setting up  (the interface to  to execute  binaries.

Installing
This part is straight forward. Run:

Once this completes, the binary should exist. You can try running on the program created above with. The program fails because it doesn't know how to find, but it wouldn't get this far if wasn't working.

That's all there is to installing for the host environment.

Setting up
Earlier, while preparing the kernel we enabled. This feature allows us to tell the kernel that we want files with certain beginning signatures to be executable, and to execute them under a specified program. There are three ways to do this, depending on your needs:

Option 1: temporarily activate by hand

This option isn't recommended as it will not survive a host system reboot, but it's good to know how to do it. To temporarily tell the kernel how to run programs, execute the following:

This will create a file the file. It is safe to this file, which will give you useful information on what has been configured.

If at any point you wish to disable, run:. The kernel will unregister the configuration and the file in that directory will disappear.

Option 2: persistently enable using OpenRC

Whether you use Option 2 or Option 3 depends on whether you are using OpenRC or systemd on the host system. If you are using OpenRC (the default for gentoo installs), start and enable the service:

After starting the services, verify that exists.

Option 3: persistently enable using systemd

If the host system uses systemd, systemd has a service called to configure binary formats.

Create a file the file containing:

Now, reload daemons and restart the service (it should already be enabled unless it's been intensionally disabled):

Again, verify that exists.

At this point, the host system is ready to start cross-compiling gentoo for the Raspberry Pi 3b.

A note on the method
Any base Gentoo system is expected to have at least the full set of ebuilds in included. The main focus of what follows is building for the Pi. There is no particular reason other packages can't be built in the cross-compilation stage, and there is good reason to do so. Cross-compilation (depending on the host system) is likely to be much faster than building either in qemu or on the target system (Pi). However, some packages fail to cross-compile correctly.

There are multiple ways out there to go about cross-compiling {{c|@system} [REFS HERE]. This method has worked well for the author with a minimum of hacking and breakage.

The method proceeds in the following steps:


 * 1) build all  packages that don't fail under crossdev, making binary packages (for all steps)
 * 2) install binary packages in a directory that will be used for qemu chroot
 * 3) build broken packages under qemu chroot
 * 4) build further dependencies under crossdev until they break
 * 5) go back to qemu chroot for broken packages, repeat until  is complete
 * 6) use binary packages to install on the Pi microSD card
 * 7) qemu chroot into the microSD card and re-install the binary packages (reasons explained later)
 * 8) configure, etc...

Depending on the user's requirements, some steps could be skipped.

Initial build with crossdev
Getting the process started for building is straight forward, but takes a while to complete. Start the build using:

This will build a lot of packages and install them under. The option tells emerge to just keep building packages even if one fails along the way (skipping other packages that depend on it). This should build the majority of packages successfully.

It probably isn't necessary to exclude at this point. Perl (and subsequent perl modules) will not cross-compile correctly. It will build, but not be functional. It's excluded here just-in-case, and as a reminder. Perl isn't actually part of the build currently.

If all went well, many of the core packages are now built and have binary packages in (it wouldn't hurt to verify that they are there).

Cross-compiling gcc
was excluded in the previous step for two reasons:
 * 1) As of the writing of this article,  will fail to build as long as the  and  parameters are present in the.
 * 2) More importantly, we are building into the same directory that our cross-compiler lives in, and building {{c|gcc} on top of it may overwrite it, and we will want the crossdev compiler functional throughout.

We need to do two things to compile {{c|gcc}. First, we need to disable and. Second, we need to be sure gcc builds as a binary package only, and does not install.

To accomplish this, run:

This will build a functional package for  without installing it. It won't be fully optimized, but as an optional later step, an optimized can be build either under  or on the native Pi install once it's complete.

The build should be complete enough at this point that it can be run under.

Creating our "build" directory
We will now create our build directory. This will mimic the final system to help build the remaining binary packages that failed.

The build directory will be assumed to be at. Create this directory.

To install the current binary packages in the build directory, run:

There should now be a mostly complete install in.

Configuring the chroot environment
[TODO]

Entering the qemu chroot
We will setup the qemu chroot exactly like any other chroot. The directory and its programs are already in place, and the kernel knows how to execute aarch64 binaries. Once the chroot is accomplished, the binaries will also be able to find their libraries. We will be entering and leaving this (and other) chroot(s) multiple times, so we'll create two scripts.

This script can be anywhere in the system and set executable. is a reasonable spot, so that it's in the execution path. Note that, among other necessary mounts, this script will make and  available within the chroot, appearing to be in the same spot.

This script needs to be placed somewhere under.

Finally, the chroot environment needs its own copy of, so copy it there:

Everything should be ready now. To enter the chroot environment, run:

You should find yourself in a new shell under the chroot. Basic utilities should be available and working.

Now, initialize the environment with:

Now should be working within the qemu chroot environment.