User:NeddySeagoon/Pi3 Build Root

Building For Memory Constrained Systems
There are several ways to build for a memory constrained system like the Raspberry Pi. * Build locally with a large swap space * Build locally with the help of cross distcc. * Build in an QEMU chroot on a completely different architecture. * Build in an QEMU chroot on a completely different architecture with the aid of cross distcc.

The first has the advantage of requiring no special setup.

The second speeds thing up considerably as building and perhaps preprocessing, are delegated to a more powerful system. The Memory Constrained system still has to do the linking and there are limits to how much swap can be used.

The third is slower as the target CPU is emulated in software by QEMU but it can use all the hosts CPU(s) and RAM. Its also possible to add cross distcc to the chroot to call the coss compiler on the chroot host or other systems.

A QEMU chroot is not a silver bullet. Not all syscalls are implemented, some missing will be tolerated others must be avoided.

Motivation
The driver to do this was Firefox-54. Firefox-54 has a hard dependency on rust. Rust comes with a bundled LLVM which adds to the burden. Rust has its own package manager, cargo, which needs rust to build.

A Raspberry Pi 3 with 1GB RAM will not build rust. No rust, no cargo, no Firefox-54 or later on a Raspberry Pi 3 or other similar aarch64 systems.

Aarch64 Metrics - Using glibc
glibc was choosen as a benchmark as it supports parallel builds well.

It appears that the chroot is about half the speed of the real hardware when its hosted on a AMD Phenom II X6 1090T.

AMD Phenom(tm) II X6 1090T Processor Six Core Native Build 3.2GHz
As a reference point. This is as good as it gets. Its both the qemu chroot host and the cross distcc helper in these benchmarks. emerge -av1 =sys-libs/glibc-2.24-r2

Sat Jun 17 17:57:13 2017 >>> sys-libs/glibc-2.24-r2 merge time: 4 minutes and 35 seconds.

qemu chroot
emerge -av1 =sys-libs/glibc-2.24-r2

Sat Jun 17 18:49:04 2017 >>> sys-libs/glibc-2.24-r2 merge time: 47 minutes and 49 seconds.

The major difference from the above is the arm64 emulation.

qemu chroot with cross distcc
The compiles are pushed over the network to 127.0.0.2, so they stand out in

Preprocessing in still carried out in the qemu chroot. In exchange for native compiling, the chroot has to drive the network.

Sat Jun 17 19:33:51 2017 >>> sys-libs/glibc-2.24-r2 merge time: 38 minutes and 28 seconds.

qemu chroot with cross distcc and pump mode
Sat Jun 17 20:51:05 2017 >>> sys-libs/glibc-2.24-r2 merge time: 37 minutes and 54 seconds.

Still with a lot of localhost preprocessing

Raspberry Pi 4 Cores at 1.2GHz
FEATURES="-distcc -distcc-pump" MAKEOPTS="-j5" emerge glibc -1av

Mon Jun 19 14:00:54 2017 >>> sys-libs/glibc-2.24-r2 merge time: 32 minutes and 18 seconds.

Raspberry Pi 4 Cores at 1.2GHz with cross distcc
FEATURES="distcc -distcc-pump" MAKEOPTS="-j5" emerge glibc -1

Mon Jun 19 14:43:47 2017 >>> sys-libs/glibc-2.24-r2 merge time: 24 minutes and 55 seconds.

and with  as that's what the helper normally runs at.

Mon Jun 19 15:49:45 2017 >>> sys-libs/glibc-2.24-r2 merge time: 23 minutes and 49 seconds.

Raspberry Pi 4 Cores at 1.2GHz with pump mode cross distcc
Now the helpers can help with preprocessing at the expense of the Pi running compression for the network link. FEATURES="distcc distcc-pump" MAKEOPTS="-j8" emerge glibc -1

Mon Jun 19 16:16:06 2017 >>> sys-libs/glibc-2.24-r2 merge time: 23 minutes and 45 seconds.

No preprocessing was observed to be sent to the helper. That probably indicates a problem on the Pi.

It looks like glibc is broken again too.

Setting Up An Aarch64 Chroot For a Pi 3
The chroot target directory is  which was provided by crossdev when the cross toolchain was built. crossdev -t aarch64-unknown-linux-gnu See the Raspberry_Pi_3_64_bit_Install guide. Unless some pure cross compiling has been performed, its almost empty. Fix the /usr/aarch64-unknown-linux-gnu/etc/portage/make.profile symlink. The default setting of embedded is not useful.

The chroot needs to be populated with arm64 packages, there are two possible sources.
 * Use existing aarch64 packages from outside the chroot, e.g. from your arm64 install.
 * Using the arm64 stage3 tarball

Using Existing Aarch64 Packages From Outside The Chroot
Binary packages have been saved to /packages on your existing arm64 install. If not, you can't use this method.

Packaging the entire install with quickpkg is possible but is left as an exercise for the reader.

Copy the existing world file  to

Copy the existing arm64  to   EXCEPT make.conf The existing /usr/aarch64-unknown-linux-gnu/etc/portage/make.conf has some cross compile settings that will be needed. Merge the existing arm64 USE, INPUT_DEVICES, VIDEO_CARDS settings and so on, into /usr/aarch64-unknown-linux-gnu/etc/portage/make.conf

Mount the existing Aarch64 packages directory to

emerge-wrapper -t aarch64-unknown-linux-gnu --init aarch64-unknown-linux-gnu-emerge -KuDNav @world Will offer to bring @world up to date using the binaries. Review the output for any changed USE flags and so on. Adjust /usr/aarch64-unknown-linux-gnu/etc/portage/make.conf as required.

Answer  when it looks good.

The command will fail if the dependency tree cannot be satisfied with the provided binaries. Either provide the binaries or change the -K to -k, which will cause portage to attempt to cross compile any required packages.

Reusing the world file
Users missing the binary packages can still start with the stage3 tarball and their world file.

Using the arm64 stage3 tarball
Users that are sill waiting for their aarch64 system to arrive can populate /usr/aarch64-unknown-linux-gnu/ with the current arm64 stage3 tarball and do a handbook install. The Handbook:AMD64 will serve as a guide as there is no arm64 handbook yet.

Do follow the steps for your host kernel and binfmt later in this guide, or will fail.

Host Kernel
See Embedded Handbook/General/Compiling with qemu user chroot

<*> Kernel support for MISC binaries

Is required. This setting has a  option too. However, switching the option on changes other parts of the kernel, so a full kernel build, install and reboot is required.

Static QEMU
QEMU will emulate the targen processor is software. Its going to be installed in /usr/aarch64-unknown-linux-gnu/. Its the only program in /usr/aarch64-unknown-linux-gnu/ that will actually run on the host. As everything else in /usr/aarch64-unknown-linux-gnu/ is built for the target CPU, QEMU cannot call any librares. It must be self contained.

This means that its a bad idea to use the chroot to emerge qemu. Add qemu to package.mask inside the chroot.

Building and Installing the Static QEMU
See Embedded Handbook/General/Compiling with qemu user chroot

binfmt service
The binfmt service allows the host to direct forigen binaries to QEMU. Its provided by sys-apps/openrc. Systemd users need to do their homework. Either add it to the default runlevel or  as required.

Preparaing to Chroot
This is best done with a script, since it needs to be performed every time the build root is used. It would be possible to add the mounts to the hosts /etc/fstab too.


 * 1) !/bin/bash


 * 1) run this script to prepare the arm64 chroot
 * 2) for QEMU building

/etc/init.d/binfmt start
 * 1) start the binfmt service on the host
 * 2) harmless warning if its already running

cd /usr/aarch64-unknown-linux-gnu/

mount --bind /usr/portage usr/portage mount --bind /usr/portage/distfiles usr/portage/distfiles mount --bind /proc proc mount --bind /sys sys mount --bind /dev dev mount --bind /dev/pts dev/pts
 * 1) not /usr/portage/packages as they are for the host!

mount tmpfs -t tmpfs -o rw,nosuid,nodev,noexec dev/shm
 * 1) must be after the bind mount of dev


 * 1) don't actually do the chroot

Do not mount the Hosts packages directory. It should be safe ...

USE Flags
app-misc/pax-utils must be built with  for use inside the chroot. There will be an unimplemented sysctl 277 error at package install time when using the chroot for builds. This will prevent packages being installed

app-misc/pax-utils will cross compile, so its no hardship to fix.

Ready Made Raspberry Pi 3 Aarch64 Stage4 chroot
Download from NeddySeagoons Pi3 64 Bit stuff.

Be sure to read the README_chroot since getting it wrong will ruin your whole day.

Its just over 4G on my filesystem.

No kernel is required as this is a chroot. The chroot host still needs kernel support and the binfmt service started.

So far, its untested since the upload. If it works for you, feedback would be appreciated if it fails, I would like to fix it.

This approach has the advantage of including rust and cargo already built.

Note: qemu should be masked in this chroot but isn't. Updating qemu will overwrite the static qemu built for the host.

Chroot
chroot /usr/aarch64-unknown-linux-gnu/ /bin/bash --login source /etc/profile export PS1="aarch64 $PS1"

If all is well, your host is running an arm64 /bin/bash with QEMU emulating an arm64. If not, you got an error message from the chroot command.

In some cases the aarch64 emulation needs to be registered with the kernel, its one single long command.

Cross Distcc
Install distcc into the aarch64 chroot in the normal way. In the  file list a helper at 127.0.0.2/8,cpp,lzo The /8 is send up to 8 jobs. cpp allows pump mode and lzo is the compression to be used with pump mode.

On the host, distcc needs to

The 127.0.0.2 address is used so that its clear in distcc-mon, what is being compiled where.

Add any more helpers you may have too.

Add distcc to FEATURES in make.conf.

DISTCC_DIR="/var/tmp/portage/.distcc/" distccmon-text 5 11249 Compile     gconv_simple.c                                127.0.0.2[5] 11337 Compile     gconv.c                                       127.0.0.2[6] 11330 Compile     gconv_open.c                                  127.0.0.2[7] 11374 Preprocess                                                localhost[0] 11386 Preprocess                                                localhost[2] 11391 Preprocess                                                localhost[3]

Objects processed by 127.0.0.2 use the cross compiler which runs natively on the host but emits code for arm64. This avoids the QEMU emulation overhead for gcc.

Cross Distcc with Pump Mode
Set up cross distcc as above,

Add distcc-pump to FEATURES in make.conf.

Xorg
The intersection of the set of GPUs that appear on aarch64 systems and the set which appear on AMD/Intel systems is the empty set. Most graphics cards in AMD/Intel systems provide a BIOS ROM and require that code from that ROM is executed to initialise the card. Aarch64 systems cannot execute that code.

It looks line an interesting challenge. To run Xorg from an emulated aarch64 system executing on an AMD/Intel platform.

Firefox
First we need rust.

Rust
Unfortunately rust needs rust to build. That's just like icedtea and gcc. To make that work the ebuild must be patched. The aarch64 support is there upstream, its just not included in the in tree ebuild.

Copy the ebuild to your overlay and apply the following patch.

--- rust-1.16.0.ebuild 2017-03-23 20:18:30.074368422 +0000 +++ rust-1.16.0-r100.ebuild    2017-06-15 19:47:05.857067998 +0100 @@ -26,6 +26,7 @@ STAGE0_VERSION="1.$(($(get_version_component_range 2) - 1)).1" RUST_STAGE0_amd64="rustc-${STAGE0_VERSION}-x86_64-unknown-linux-gnu" RUST_STAGE0_x86="rustc-${STAGE0_VERSION}-i686-unknown-linux-gnu" +RUST_STAGE0_arm64="rustc-${STAGE0_VERSION}-aarch64-unknown-linux-gnu"

DESCRIPTION="Systems programming language from Mozilla" HOMEPAGE="http://www.rust-lang.org/" @@ -33,6 +34,7 @@ SRC_URI="https://static.rust-lang.org/dist/${SRC} -> rustc-${PV}-src.tar.gz       amd64? ( https://static.rust-lang.org/dist/${RUST_STAGE0_amd64}.tar.gz )        x86? ( https://static.rust-lang.org/dist/${RUST_STAGE0_x86}.tar.gz ) +	arm64? ( https://static.rust-lang.org/dist/${RUST_STAGE0_arm64}.tar.gz ) "

LICENSE="|| ( MIT Apache-2.0 ) BSD-1 BSD-2 BSD-4 UoI-NCSA" Don't forget to make the digest.

It needs to be ~arm64 keyworded too, or added to package.accept_keywords.

Cargo
Cargo is much the same. Aarch64 support is available. However, the 2016-09-01 nghtly snapshot, which is what the ebuild would fetch, segfaults in the QEMU chroot. The 2016-11-28 version seems to build.

Copy the ebuild to your overlay and apply the following patch. --- cargo-0.17.0.ebuild	2017-03-18 10:41:17.000000000 +0000 +++ /root/cargo-0.17.0-r100.ebuild	2017-06-18 21:04:41.164444192 +0100 @@ -3,7 +3,7 @@ EAPI=6 -CARGO_SNAPSHOT_DATE="2016-09-01" +CARGO_SNAPSHOT_DATE="2016-11-28" CRATES=" advapi32-sys-0.2.0 aho-corasick-0.5.3 @@ -93,12 +93,16 @@ 	amd64? ( 		https://static.rust-lang.org/cargo-dist/${CARGO_SNAPSHOT_DATE}/cargo-nightly-x86_64-unknown-linux-gnu.tar.gz -> 		cargo-snapshot-amd64-${CARGO_SNAPSHOT_DATE}.tar.gz +	) +	arm64? ( +               https://static.rust-lang.org/cargo-dist/${CARGO_SNAPSHOT_DATE}/cargo-nightly-aarch64-unknown-linux-gnu.tar.gz -> +                cargo-snapshot-aarch64-${CARGO_SNAPSHOT_DATE}.tar.gz 	)" RESTRICT="mirror" LICENSE="|| ( MIT Apache-2.0 )" SLOT="0" -KEYWORDS="~amd64 ~x86" +KEYWORDS="~amd64 ~arm64 ~x86" IUSE="doc libressl" @@ -126,6 +130,7 @@ 	# where could be 'x86_64' (amd64) or 'i686' (x86) use amd64 && CTARGET="x86_64-unknown-linux-gnu" use x86 && CTARGET="i686-unknown-linux-gnu" +	use arm64 && CTARGET="aarch64-unknown-linux-gnu" # NOTE: 'disable-nightly' is used by crates (such as 'matches') to entirely # skip their internal libraries that make use of unstable rustc features.

Firefox-54
With rust and cargo in packages, firefox itself might build on the Pi with help from cross distcc. (it won't, not even at -j1).

Rust is profile masked on arm64 but its a hard dependency for Firefox-54.

Continue to build firefox in the QEMU chroot

If the build fails with

File "/usr/lib64/python2.7/multiprocessing/synchronize.py", line 75, in __init__ sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue) OSError: [Errno 38] Function not implemented this indicates that /dev/shm is not available in the chroot.

will fix it.

Here, the build ends with >>> Install firefox-54.0 into /var/tmp/portage/www-client/firefox-54.0/image/ category www-client * PT_PAX marking -m /var/tmp/portage/www-client/firefox-54.0/work/firefox-54.0/ff/dist/bin/xpcshell with scanelf * XATTR_PAX marking -me /var/tmp/portage/www-client/firefox-54.0/work/firefox-54.0/ff/dist/bin/xpcshell with setfattr * Adding prefs from mozconfig to /var/tmp/portage/www-client/firefox-54.0/work/firefox-54.0/ff/dist/bin/browser/defaults/preferences/all-gentoo.js make -j8 DESTDIR=/var/tmp/portage/www-client/firefox-54.0/image/ install make[1]: Entering directory '/var/tmp/portage/www-client/firefox-54.0/work/firefox-54.0/ff/browser/installer' which is the stalled optimise seen in versions of firefox>=52 on arm64.

There is a patch for firefox, create   with the following content --- ./browser/config/mozconfigs/linux32/common-opt.orig	2017-03-27 16:47:57.039428674 -0700 +++ ./browser/config/mozconfigs/linux32/common-opt	2017-03-27 16:43:26.004591584 -0700 @@ -3,6 +3,7 @@ ac_add_options --enable-update-channel=${MOZ_UPDATE_CHANNEL} ac_add_options --with-google-api-keyfile=/builds/gapi.data ac_add_options --with-mozilla-api-keyfile=/builds/mozilla-desktop-geoloc-api.key +ac_add_options --disable-startupcache . $topsrcdir/build/unix/mozconfig.linux32 --- ./browser/config/mozconfigs/linux64/common-opt.orig	2017-03-27 16:46:54.794982020 -0700 +++ ./browser/config/mozconfigs/linux64/common-opt	2017-03-27 16:47:05.345170624 -0700 @@ -3,6 +3,7 @@ ac_add_options --enable-update-channel=${MOZ_UPDATE_CHANNEL} ac_add_options --with-google-api-keyfile=/builds/gapi.data ac_add_options --with-mozilla-api-keyfile=/builds/mozilla-desktop-geoloc-api.key +ac_add_options --disable-startupcache . $topsrcdir/build/unix/mozconfig.linux note that the patch ends with a blank line

This patch alone does not work for me, it also takes FEATURES="-sandbox -usersandbox" MAKEOPTS="-j8" emerge firefox

Next step, install on the Pi

Chroot Build Times
aarch64 NeddySeagoon_Static / # genlop -t rust cargo firefox * dev-lang/rust

Sun Jun 18 04:30:42 2017 >>> dev-lang/rust-1.16.0-r100 merge time: 7 hours, 8 minutes and 6 seconds.

* dev-util/cargo

Sun Jun 18 10:07:42 2017 >>> dev-util/cargo-0.17.0 merge time: 36 minutes and 7 seconds.

* www-client/firefox

Tue Jun 20 21:14:21 2017 >>> www-client/firefox-54.0 merge time: 2 hours, 51 minutes and 8 seconds.

Install the Packages on the Target (Theory)
Move the packages and the ebuilds that they were built with to the target and emerge the binaries. These are the packages that would be merged, in order:

Calculating dependencies... done! [binary N     ] dev-util/cargo-0.17.0-r100::gentoo  USE="-debug (-doc) (-libressl)" 0 KiB [binary N     ] dev-lang/rust-1.16.0-r100:stable/1.16::gentoo  USE="-clang -debug (-doc) -libcxx" 0 KiB [binary    U  ] www-client/firefox-54.0::gentoo [51.0.1::gentoo] USE=" ..."

Careful readers will note that the writer did not use an overlay for the patched ebuilds.