User:NeddySeagoon/Pi3 Build Root

From Gentoo Wiki
Jump to: navigation, search

Building For Memory Constrained Systems

There are several ways to build for a memory constrained system like the Raspberry Pi.

* Build locally with a large swap space
* Build locally with the help of cross distcc.
* Build in an QEMU chroot on a completely different architecture.
* Build in an QEMU chroot on a completely different architecture with the aid of cross distcc.

The first has the advantage of requiring no special setup.

The second speeds thing up considerably as building and perhaps preprocessing, are delegated to a more powerful system. The Memory Constrained system still has to do the linking and there are limits to how much swap can be used.

The third is slower as the target CPU is emulated in software by QEMU but it can use all the hosts CPU(s) and RAM. Its also possible to add cross distcc to the chroot to call the coss compiler on the chroot host or other systems.

A QEMU chroot is not a silver bullet. Not all syscalls are implemented, some missing will be tolerated others must be avoided.


The driver to do this was Firefox-54. Firefox-54 has a hard dependency on rust. Rust comes with a bundled LLVM which adds to the burden. Rust has its own package manager, cargo, which needs rust to build.

A Raspberry Pi 3 with 1GB RAM will not build rust. No rust, no cargo, no Firefox-54 or later on a Raspberry Pi 3 or other similar aarch64 systems.

Aarch64 Metrics - Using glibc

glibc was choosen as a benchmark as it supports parallel builds well.

It appears that the chroot is about half the speed of the real hardware when its hosted on a AMD Phenom II X6 1090T.

AMD Phenom(tm) II X6 1090T Processor Six Core Native Build 3.2GHz

As a reference point. This is as good as it gets. Its both the qemu chroot host and the cross distcc helper in these benchmarks.

emerge -av1 =sys-libs/glibc-2.24-r2
    Sat Jun 17 17:57:13 2017 >>> sys-libs/glibc-2.24-r2
      merge time: 4 minutes and 35 seconds.

qemu chroot

emerge -av1 =sys-libs/glibc-2.24-r2
    Sat Jun 17 18:49:04 2017 >>> sys-libs/glibc-2.24-r2
      merge time: 47 minutes and 49 seconds.

The major difference from the above is the arm64 emulation.

qemu chroot with cross distcc

The compiles are pushed over the network to, so they stand out inDISTCC_DIR="/var/tmp/portage/.distcc/" distccmon-text 5

Preprocessing is still carried out in the qemu chroot. In exchange for native compiling, the chroot has to drive the network.

FEATURES=distcc emerge -av1 =sys-libs/glibc-2.24-r2

    Sat Jun 17 19:33:51 2017 >>> sys-libs/glibc-2.24-r2
      merge time: 38 minutes and 28 seconds.

qemu chroot with cross distcc and pump mode

FEATURES="distcc distcc-pump" emerge -av1 =sys-libs/glibc-2.24-r2

    Sat Jun 17 20:51:05 2017 >>> sys-libs/glibc-2.24-r2
      merge time: 37 minutes and 54 seconds.

Still with a lot of localhost preprocessing

Raspberry Pi 4 Cores at 1.2GHz

FEATURES="-distcc -distcc-pump" MAKEOPTS="-j5" emerge glibc -1av
    Mon Jun 19 14:00:54 2017 >>> sys-libs/glibc-2.24-r2
      merge time: 32 minutes and 18 seconds.

Raspberry Pi 4 Cores at 1.2GHz with cross distcc

FEATURES="distcc -distcc-pump" MAKEOPTS="-j5" emerge glibc -1
    Mon Jun 19 14:43:47 2017 >>> sys-libs/glibc-2.24-r2
      merge time: 24 minutes and 55 seconds.

and with MAKEOPTS="-j8" as that's what the helper normally runs at.

    Mon Jun 19 15:49:45 2017 >>> sys-libs/glibc-2.24-r2
      merge time: 23 minutes and 49 seconds.

Raspberry Pi 4 Cores at 1.2GHz with pump mode cross distcc

Now the helpers can help with preprocessing at the expense of the Pi running compression for the network link.

FEATURES="distcc distcc-pump" MAKEOPTS="-j8" emerge glibc -1
    Mon Jun 19 16:16:06 2017 >>> sys-libs/glibc-2.24-r2
      merge time: 23 minutes and 45 seconds.

No preprocessing was observed to be sent to the helper. That probably indicates a problem on the Pi.

Setting Up An Aarch64 Chroot For a Pi 3

The chroot target directory is /usr/aarch64-unknown-linux-gnu/ which was provided by crossdev when the cross toolchain was built.

crossdev -t aarch64-unknown-linux-gnu

See the Raspberry_Pi_3_64_bit_Install guide. Unless some pure cross compiling has been performed, its almost empty. Fix the /usr/aarch64-unknown-linux-gnu/etc/portage/make.profile symlink. The default setting of embedded is not useful.

The chroot needs to be populated with arm64 packages, there are two possible sources.

  • Use existing aarch64 packages from outside the chroot, e.g. from your arm64 install.
  • Using the arm64 stage3 tarball

Using Existing Aarch64 Packages From Outside The Chroot

Binary packages have been saved to /packages on your existing arm64 install. If not, you can't use this method.

Packaging the entire install with quickpkg is possible but is left as an exercise for the reader.

Copy the existing world file /var/lib/portage/world to /usr/aarch64-unknown-linux-gnu/var/lib/portage/world

Copy the existing arm64 /etc/portage/ to /usr/aarch64-unknown-linux-gnu/etc/portage/ EXCEPT make.conf The existing /usr/aarch64-unknown-linux-gnu/etc/portage/make.conf has some cross compile settings that will be needed. Merge the existing arm64 USE, INPUT_DEVICES, VIDEO_CARDS settings and so on, into /usr/aarch64-unknown-linux-gnu/etc/portage/make.conf

Mount the existing Aarch64 packages directory to /usr/aarch64-unknown-linux-gnu/packages

emerge-wrapper -t aarch64-unknown-linux-gnu --init
aarch64-unknown-linux-gnu-emerge -KuDNav @world

Will offer to bring @world up to date using the binaries. Review the output for any changed USE flags and so on. Adjust /usr/aarch64-unknown-linux-gnu/etc/portage/make.conf as required.

Answer y when it looks good.

The command will fail if the dependency tree cannot be satisfied with the provided binaries. Either provide the binaries or change the -K to -k, which will cause portage to attempt to cross compile any required packages.

Reusing the world file

Users missing the binary packages can still start with the stage3 tarball and their world file.

Using the arm64 stage3 tarball

Users that are sill waiting for their aarch64 system to arrive can populate /usr/aarch64-unknown-linux-gnu/ with the current arm64 stage3 tarball and do a handbook install. The Handbook:AMD64 will serve as a guide as there is no arm64 handbook yet.

Do follow the steps for your host kernel and binfmt later in this guide, or chroot /usr/aarch64-unknown-linux-gnu/ /bin/bash --login will fail.

Host Kernel

See Embedded Handbook/General/Compiling with qemu user chroot

<*> Kernel support for MISC binaries

Is required. This setting has a <M> option too. However, switching the option on changes other parts of the kernel, so a full kernel build, install and reboot is required.

Static QEMU

QEMU will emulate the target processor in software. Its going to be installed in /usr/aarch64-unknown-linux-gnu/. It is the only program in /usr/aarch64-unknown-linux-gnu/ that will actually run on the host. As everything else in /usr/aarch64-unknown-linux-gnu/ is built for the target CPU, QEMU cannot call any librares. It must be self contained.

This means that its a bad idea to use the chroot to emerge qemu. Add qemu to package.mask inside the chroot.

Building and Installing the Static QEMU

See Embedded Handbook/General/Compiling with qemu user chroot

binfmt service

The binfmt service allows the host to direct forigen binaries to QEMU. Its provided by sys-apps/openrc. Systemd users need to do their homework. Either add it to the default runlevel or /etc/init.d/binfmt start as required.

Preparaing to Chroot

This is best done with a script, since it needs to be performed every time the build root is used. It would be possible to add the mounts to the hosts /etc/fstab too.


# run this script to prepare the arm64 chroot
# for QEMU building

# start the binfmt service on the host
# harmless warning if its already running
/etc/init.d/binfmt start

cd /usr/aarch64-unknown-linux-gnu/

mount --bind /usr/portage usr/portage
mount --bind /usr/portage/distfiles usr/portage/distfiles
# not /usr/portage/packages as they are for the host!
mount --bind /proc proc
mount --bind /sys sys
mount --bind /dev dev
mount --bind /dev/pts dev/pts 

# must be after the bind mount of dev
mount tmpfs -t tmpfs -o rw,nosuid,nodev,noexec dev/shm

# don't actually do the chroot

Do not mount the Hosts packages directory. It should be safe ...

USE Flags

app-misc/pax-utils must be built with USE=-seccomp for use inside the chroot. There will be an unimplemented sysctl 277 error at package install time when using the chroot for builds. This will prevent packages being installed

app-misc/pax-utils will cross compile, so its no hardship to fix.

Ready Made Raspberry Pi 3 Aarch64 Stage4 chroot

Download from NeddySeagoons Pi3 64 Bit stuff.

Be sure to read the README_chroot since getting it wrong will ruin your whole day.

Its just over 4G on my filesystem.

No kernel is required as this is a chroot.

The chroot host still needs kernel support and the binfmt service started.

So far, its untested since the upload. If it works for you, feedback would be appreciated if it fails, I would like to fix it.

This approach has the advantage of including rust and cargo already built.

Note: qemu should be masked in this chroot but isn't. Updating qemu will overwrite the static qemu built for the host.


chroot /usr/aarch64-unknown-linux-gnu/ /bin/bash --login
source /etc/profile
export PS1="aarch64 $PS1"

If all is well, your host is running an arm64 /bin/bash with QEMU emulating an arm64. If not, you got an error message from the chroot command.

In some cases the aarch64 emulation needs to be registered with the kernel, its one single long command.

echo ':aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-aarch64:' > /proc/sys/fs/binfmt_misc/register

Loose Ends

Cross Distcc

Install distcc into the aarch64 chroot in the normal way. In the /etc/distcc/hosts file list a helper at,cpp,lzo The /8 is send up to 8 jobs. cpp allows pump mode and lzo is the compression to be used with pump mode.

On the host, distcc needs to --allow

The address is used so that its clear in distcc-mon, what is being compiled where.

Add any more helpers you may have too.

Add distcc to FEATURES in make.conf.

DISTCC_DIR="/var/tmp/portage/.distcc/" distccmon-text 5
11249  Compile     gconv_simple.c                      [5]
11337  Compile     gconv.c                             [6]
11330  Compile     gconv_open.c                        [7]
11374  Preprocess                                                localhost[0]
11386  Preprocess                                                localhost[2]
11391  Preprocess                                                localhost[3]

Objects processed by use the cross compiler which runs natively on the host but emits code for arm64. This avoids the QEMU emulation overhead for gcc.

Cross Distcc with Pump Mode

Set up cross distcc as above,

Add distcc-pump to FEATURES in make.conf.


The intersection of the set of GPUs that appear on aarch64 systems and the set which appear on AMD/Intel systems is the empty set. Most graphics cards in AMD/Intel systems provide a BIOS ROM and require that code from that ROM is executed to initialise the card. Aarch64 systems cannot execute that code.

It looks line an interesting challenge. To run Xorg from an emulated aarch64 system executing on an AMD/Intel platform.

Firefox >=54.0

First we need rust.


Unfortunately rust needs rust to build. That's just like icedtea and gcc. To make that work the ebuild must be patched. The aarch64 support is there upstream, its just not included in the in tree ebuild.

Copy the ebuild to your overlay and apply the following patch.

--- rust-1.16.0.ebuild  2017-03-23 20:18:30.074368422 +0000
+++ rust-1.16.0-r100.ebuild     2017-06-15 19:47:05.857067998 +0100
@@ -26,6 +26,7 @@
 STAGE0_VERSION="1.$(($(get_version_component_range 2) - 1)).1"

 DESCRIPTION="Systems programming language from Mozilla"
@@ -33,6 +34,7 @@
 SRC_URI="${SRC} -> rustc-${PV}-src.tar.gz
        amd64? (${RUST_STAGE0_amd64}.tar.gz )
        x86? (${RUST_STAGE0_x86}.tar.gz )
+	arm64? (${RUST_STAGE0_arm64}.tar.gz )

 LICENSE="|| ( MIT Apache-2.0 ) BSD-1 BSD-2 BSD-4 UoI-NCSA"

Don't forget to make the digest.

It needs to be ~arm64 keyworded too, or added to package.accept_keywords.


Cargo is much the same. Aarch64 support is available. However, the 2016-09-01 nghtly snapshot, which is what the ebuild would fetch, segfaults in the QEMU chroot. The 2016-11-28 version seems to build.

Copy the ebuild to your overlay and apply the following patch.

--- cargo-0.17.0.ebuild	2017-03-18 10:41:17.000000000 +0000
+++ /root/cargo-0.17.0-r100.ebuild	2017-06-18 21:04:41.164444192 +0100
@@ -3,7 +3,7 @@
@@ -93,12 +93,16 @@
 	amd64? (${CARGO_SNAPSHOT_DATE}/cargo-nightly-x86_64-unknown-linux-gnu.tar.gz ->
+	)
+	arm64? (
+      ${CARGO_SNAPSHOT_DATE}/cargo-nightly-aarch64-unknown-linux-gnu.tar.gz ->
+                cargo-snapshot-aarch64-${CARGO_SNAPSHOT_DATE}.tar.gz
 LICENSE="|| ( MIT Apache-2.0 )"
-KEYWORDS="~amd64 ~x86"
+KEYWORDS="~amd64 ~arm64 ~x86"
 IUSE="doc libressl"
@@ -126,6 +130,7 @@
 	# where <arch> could be 'x86_64' (amd64) or 'i686' (x86)
 	use amd64 && CTARGET="x86_64-unknown-linux-gnu"
 	use x86 && CTARGET="i686-unknown-linux-gnu"
+	use arm64 && CTARGET="aarch64-unknown-linux-gnu"
 	# NOTE: 'disable-nightly' is used by crates (such as 'matches') to entirely
 	# skip their internal libraries that make use of unstable rustc features.


With rust and cargo in packages, firefox itself might build on the Pi with help from cross distcc. (it won't, not even at -j1).

Rust is profile masked on arm64 but its a hard dependency for Firefox-54.

Continue to build firefox in the QEMU chroot

If the build fails with

File "/usr/lib64/python2.7/multiprocessing/", line 75, in __init__
    sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
 OSError: [Errno 38] Function not implemented

this indicates that /dev/shm is not available in the chroot.

mount tmpfs -t tmpfs -o rw,nosuid,nodev,noexec /dev/shm will fix it.

Here, the build ends with

 >>> Install firefox-54.0 into /var/tmp/portage/www-client/firefox-54.0/image/ category www-client
 * PT_PAX marking -m /var/tmp/portage/www-client/firefox-54.0/work/firefox-54.0/ff/dist/bin/xpcshell with scanelf
 * XATTR_PAX marking -me /var/tmp/portage/www-client/firefox-54.0/work/firefox-54.0/ff/dist/bin/xpcshell with setfattr
 * Adding prefs from mozconfig to /var/tmp/portage/www-client/firefox-54.0/work/firefox-54.0/ff/dist/bin/browser/defaults/preferences/all-gentoo.js
make -j8 DESTDIR=/var/tmp/portage/www-client/firefox-54.0/image/ install 
make[1]: Entering directory '/var/tmp/portage/www-client/firefox-54.0/work/firefox-54.0/ff/browser/installer'

which is the stalled optimise seen in versions of firefox>=52 on arm64.

There is a patch for firefox, create /etc/portage/patches/www-client/firefox/no-optimise.patch with the following content

--- ./browser/config/mozconfigs/linux32/common-opt.orig	2017-03-27 16:47:57.039428674 -0700
+++ ./browser/config/mozconfigs/linux32/common-opt	2017-03-27 16:43:26.004591584 -0700
@@ -3,6 +3,7 @@
 ac_add_options --enable-update-channel=${MOZ_UPDATE_CHANNEL}
 ac_add_options --with-google-api-keyfile=/builds/
 ac_add_options --with-mozilla-api-keyfile=/builds/mozilla-desktop-geoloc-api.key
+ac_add_options --disable-startupcache
 . $topsrcdir/build/unix/mozconfig.linux32
--- ./browser/config/mozconfigs/linux64/common-opt.orig	2017-03-27 16:46:54.794982020 -0700
+++ ./browser/config/mozconfigs/linux64/common-opt	2017-03-27 16:47:05.345170624 -0700
@@ -3,6 +3,7 @@
 ac_add_options --enable-update-channel=${MOZ_UPDATE_CHANNEL}
 ac_add_options --with-google-api-keyfile=/builds/
 ac_add_options --with-mozilla-api-keyfile=/builds/mozilla-desktop-geoloc-api.key
+ac_add_options --disable-startupcache
 . $topsrcdir/build/unix/mozconfig.linux

note that the patch ends with a blank line

This patch alone does not work for me, it also takes

FEATURES="-sandbox -usersandbox" MAKEOPTS="-j8" emerge firefox

Next step, install on the Pi

Chroot Build Times
aarch64 NeddySeagoon_Static / # genlop -t rust cargo firefox
 * dev-lang/rust

     Sun Jun 18 04:30:42 2017 >>> dev-lang/rust-1.16.0-r100
       merge time: 7 hours, 8 minutes and 6 seconds.

 * dev-util/cargo

     Sun Jun 18 10:07:42 2017 >>> dev-util/cargo-0.17.0
       merge time: 36 minutes and 7 seconds.

 * www-client/firefox

     Tue Jun 20 21:14:21 2017 >>> www-client/firefox-54.0
       merge time: 2 hours, 51 minutes and 8 seconds.

Install the Packages on the Target

Move the packages and the ebuilds that they were built with to the target and emerge the binaries. emerge -kav firefox

These are the packages that would be merged, in order:

Calculating dependencies... done!
[binary  N     ] dev-util/cargo-0.17.0-r100::gentoo  USE="-debug (-doc) (-libressl)" 0 KiB
[binary  N     ] dev-lang/rust-1.16.0-r100:stable/1.16::gentoo  USE="-clang -debug (-doc) -libcxx" 0 KiB
[binary     U  ] www-client/firefox-54.0::gentoo [51.0.1::gentoo] USE=" ..." 

Careful readers will note that the writer did not use an overlay for the patched ebuilds.

Complete Aarch64 KVM on an amd64 Host

It should work but this is left as an exercise to the reader.

The boot loader will probably need to be u-boot, since it will need to run on the arm64 emulator and neither grub, nor the Raspberry Pi bootcode.bin will do that.

Hint: u-boot can be used to boot the Pi. bootcode.bin is then used to load u-boot and u-boot does the rest.