User:NeddySeagoon/Pi3 Build Root

Building For Memory Constrained Systems
There are several ways to build for a memory constrained system like the Raspberry Pi. * Build locally with a large swap space * Build locally with the help of cross distcc. * Build in an QEMU chroot on a completely different architecture. * Build in an QEMU chroot on a completely different architecture with the aid of cross distcc.

The first has the advantage of requiring no special setup.

The second speeds thing up considerably as building and perhaps preprocessing, are delegated to a more powerful system. The Memory Constrained system still has to do the linking and there are limits to how much swap can be used.

The third is slower as the target CPU is emulated in software by QEMU but it can use all the hosts CPU(s) and RAM. Its also possible to add cross distcc to the chroot to call the coss compiler on the chroot host or other systems.

A QEMU chroot is not a silver bullet. Not all syscalls are implemented, some missing will be tolerated others must be avoided.

Motivation
The driver to do this was Firefox-54. Firefox-54 has a hard dependency on rust. Rust comes with a bundled LLVM which adds to the burden. Rust has its own package manager, cargo, which needs rust to build.

A Raspberry Pi 3 with 1GB RAM will not build rust. No rust, no cargo, no Firefox-54 or later on a Raspberry Pi 3 or other similar aarch64 systems.

Aarch64 Metrics - Using glibc
glibc was choosen as a benchmark as it supports parallel builds well

AMD Phenom(tm) II X6 1090T Processor Six Core Native Build 3.2GHz
As a reference point. This is as good as it gets. Its both the qemu chroot host and the cross distcc helper in these benchmarks. emerge -av1 =sys-libs/glibc-2.24-r2

Sat Jun 17 17:57:13 2017 >>> sys-libs/glibc-2.24-r2 merge time: 4 minutes and 35 seconds.

qemu chroot
emerge -av1 =sys-libs/glibc-2.24-r2

Sat Jun 17 18:49:04 2017 >>> sys-libs/glibc-2.24-r2 merge time: 47 minutes and 49 seconds.

The major difference from the above is the arm64 emulation.

qemu chroot with cross distcc
The compiles are pushed over the network to 127.0.0.2, so they stand out in

Preprocessing in still carried out in the qemu chroot. In exchange for native compiling, the chroot has to drive the network.

Sat Jun 17 19:33:51 2017 >>> sys-libs/glibc-2.24-r2 merge time: 38 minutes and 28 seconds.

did it break the chroot???????? Reverting to glibc-2.24-r1 fixed it

qemu chroot with cross distcc and pump mode
Sat Jun 17 20:51:05 2017 >>> sys-libs/glibc-2.24-r2 merge time: 37 minutes and 54 seconds.

Still with a lot of localhost preprocessing

Setting Up An Aarch64 Chroot For a Pi 3
The chroot target directory is /usr/aarch64-unknown-linux-gnu/ that was provided by crossdev when the cross toolchain was built. crossdev -t aarch64-unknown-linux-gnu See the Raspberry_Pi_3_64_bit_Install guide. Unless some pure cross compiling has been performed, its almost empty. Fix the /usr/aarch64-unknown-linux-gnu/etc/portage/make.profile symlink. The default setting of embedded is not useful.

The chroot needs to be populated with arm64 packages, there are two possible sources.
 * Use existing aarch64 packages from outside the chroot, e.g. from your arm64 install.
 * Using the arm64 stage3 tarball

Using Existing Aarch64 Packages From Outside The Chroot
Binary packages have been saved to /packages on your existing arm64 install. If not, you can't use this method.

Packaging the entire install with quickpkg is possible but is left as an exercise for the reader.

Copy the existing world file /var/lib/portage/world to /usr/aarch64-unknown-linux-gnu/var/lib/portage/world

Copy the existing arm64 /etc/portage to /usr/aarch64-unknown-linux-gnu/etc/portage EXCEPT make.conf The existing /usr/aarch64-unknown-linux-gnu/etc/portage/make.conf has some cross compile settings that will be needed. Merge the existing arm64 USE, INPUT_DEVICES, VIDEO_CARDS settings and so on, into /usr/aarch64-unknown-linux-gnu/etc/portage/make.conf

Mount the existing Aarch64 packages to /usr/aarch64-unknown-linux-gnu/packages

emerge-wrapper -t aarch64-unknown-linux-gnu --init aarch64-unknown-linux-gnu-emerge -KuDNav @world This will offer to bring @world up to date using the binaries. Review the output for any changed USE flags and so on. Adjust /usr/aarch64-unknown-linux-gnu/etc/portage/make.conf as required.

Answer y when it looks good.

The command will fail if the dependency tree cannot be satisfied with the provided binaries. Either provide the binaries or change the -K to -k, which will cause portage to attempt to cross compile any required packages.

Reusing the world file
Users missing the binary packages can still start with the stage3 tarball and their world file.

Using the arm64 stage3 tarball
Users that are sill waiting for their aarch64 system to arrive can populate /usr/aarch64-unknown-linux-gnu/ with the current arm64 stage3 tarball and do a handbook install. Do follow the steps for your host kernel and binfmt later in this guide, or will fail.

Static QEMU
QEMU will emulate the targen processor is software. Its going to be installed in /usr/aarch64-unknown-linux-gnu/. Its the only program in /usr/aarch64-unknown-linux-gnu/ that will actually run on the host. As everything else in /usr/aarch64-unknown-linux-gnu/ is built for the target CPU, QEMU cannot call any librares. It must be self contained.

Building and Installing the Static QEMU
See Embedded Handbook/General/Compiling with qemu user chroot

binfmt service
The binfmt service allows the host to direct forigen binaries to QEMU. Its provided by sys-apps/openrc. Systemd users need to do their homework. Either add it to the default runlevel or  as required.

Preparaing to Chroot
mount /dev mount /proc mount /sys mount the gentoo repository mount distfiles Do not mount the Hosts packages directory.

USE Flags
In the chroot app-misc/pax-utils must be built with USE=-seccomp. There will be an unimplemeted sysctrl 277 error at package install time when using the chroot for builds.

Ready Made Aarch64 Stage4 chroot
Download from http://bloodnoc.org/~roy/...

The host still needs kernel suppor and the binfmt service started.

Chroot
chroot /usr/aarch64-unknown-linux-gnu/ /bin/bash --login source /etc/profile mount tmpfs -t tmpfs -o rw,nosuid,nodev,noexec /dev/shm export PS1="aarch64 $PS1"

/dev was bind mounted but we need /dev/shm inside the chroot, so it needs to be mounted separately.

If all is well, your host is running an arm64 /bin/bash with QEMU emulating an arm64. If not, you got an error message.

Cross Distcc
Install distcc into the aarch64 chroot in the normal way. In the /etc/distcc/hosts file list a helper at 127.0.0.2/8,cpp,lzo The /8 is send up to 8 jobs. cpp allows pump mode and lzo is the compression to be used with pump mode.

On the host, distcc needs to --allow 127.0.0.0/8

The 127.0.0.2 address is used so that its clear in distcc-mon, what is being compiled where. Add any more helpers you may have too. Add distcc to FEATURES in make.conf.

The 127.0.0.2 address is used so that its clear in distcc-mon, what is being compiled where. Add any more helpers you may have too. Add distcc to FEATURES in make.conf.

DISTCC_DIR="/var/tmp/portage/.distcc/" distccmon-text 5 11249 Compile     gconv_simple.c                                127.0.0.2[5] 11337 Compile     gconv.c                                       127.0.0.2[6] 11330 Compile     gconv_open.c                                  127.0.0.2[7] 11374 Preprocess                                                localhost[0] 11386 Preprocess                                                localhost[2] 11391 Preprocess                                                localhost[3]

Objects processed by 127.0.0.2 use the cross compiler which runs natively on the host but emits code for arm64. This avoids the emulation overhead for gcc.

Cross Distcc with Pump Mode
Set up cross distcc as above Add distcc-pump to FEATURES in make.conf.

Firefox
First we need rust.

Rust
Unfortunately rust needs rust to build. That's just like icedtea and gcc. To make that work the ebuild must be patched. The aarch64 support is there upstream, its just not included in the in tree ebuild

--- rust-1.16.0.ebuild 2017-03-23 20:18:30.074368422 +0000 +++ rust-1.16.0-r100.ebuild    2017-06-15 19:47:05.857067998 +0100 @@ -26,6 +26,7 @@ STAGE0_VERSION="1.$(($(get_version_component_range 2) - 1)).1" RUST_STAGE0_amd64="rustc-${STAGE0_VERSION}-x86_64-unknown-linux-gnu" RUST_STAGE0_x86="rustc-${STAGE0_VERSION}-i686-unknown-linux-gnu" +RUST_STAGE0_arm64="rustc-${STAGE0_VERSION}-aarch64-unknown-linux-gnu"

DESCRIPTION="Systems programming language from Mozilla" HOMEPAGE="http://www.rust-lang.org/" @@ -33,6 +34,7 @@ SRC_URI="https://static.rust-lang.org/dist/${SRC} -> rustc-${PV}-src.tar.gz       amd64? ( https://static.rust-lang.org/dist/${RUST_STAGE0_amd64}.tar.gz )        x86? ( https://static.rust-lang.org/dist/${RUST_STAGE0_x86}.tar.gz ) +	arm64? ( https://static.rust-lang.org/dist/${RUST_STAGE0_arm64}.tar.gz ) "

LICENSE="|| ( MIT Apache-2.0 ) BSD-1 BSD-2 BSD-4 UoI-NCSA"

It needs to be ~arm64 keyworded too, or added to package.accept_keywords.

Cargo
Cargo is much the same. Aarch64 support is available. However, the 2016-09-01 nghtly snapshot, which is what the ebuild would fetch, segfaults in the QEMU c$ The 2016-11-28 version seems to build.

--- cargo-0.17.0.ebuild 2017-03-23 20:18:30.221367817 +0000 +++ cargo-0.17.0-r100.ebuild   2017-06-18 11:11:56.045854038 +0100 @@ -3,7 +3,7 @@

EAPI=6

-CARGO_SNAPSHOT_DATE="2016-09-01" +CARGO_SNAPSHOT_DATE="2016-11-28" CRATES=" advapi32-sys-0.2.0 aho-corasick-0.5.3 @@ -93,12 +93,16 @@       amd64? (                https://static.rust-lang.org/cargo-dist/${CARGO_SNAPSHOT_DATE}/cargo-nightly-x86_64-unknown-linux-gnu.tar.gz ->                cargo-snapshot-amd64-${CARGO_SNAPSHOT_DATE}.tar.gz +	)

RESTRICT="mirror" LICENSE="|| ( MIT Apache-2.0 )" SLOT="0" -KEYWORDS="~amd64 ~x86" +KEYWORDS="~amd64 ~arm64 ~x86"

IUSE="doc libressl"

@@ -126,6 +130,7 @@       # where could be 'x86_64' (amd64) or 'i686' (x86) use amd64 && CTARGET="x86_64-unknown-linux-gnu" use x86 && CTARGET="i686-unknown-linux-gnu" +	use arm64 && CTARGET="aarch64-unknown-linux-gnu"

# NOTE: 'disable-nightly' is used by crates (such as 'matches') to entirely # skip their internal libraries that make use of unstable rustc features.

Firefox-54
With rust and cargo in place, firefox itself might build. Notice the USE=(-rust%) in firefox but it wants rust anyway. Rust is hard masked on arm64.

emerge -kav firefox

* IMPORTANT: 6 news items need reading for repository 'gentoo'. * Use eselect news read to view new items.

These are the packages that would be merged, in order:

Calculating dependencies... done! [binary N     ] dev-util/cargo-0.17.0::gentoo  USE="-debug (-doc) (-libressl)" 0 KiB [binary N     ] dev-lang/rust-1.16.0-r100:stable/1.16::gentoo  USE="-clang -debug (-doc) -libcxx" 0 KiB [ebuild    U *] www-client/firefox-54.0-r100::gentoo [51.0.1::gentoo] USE="bindist dbus hwaccel jemalloc startup-notification system-harfbuzz system-icu sy$ -custom-optimization -debug -gmp-autoupdate (-hardened) -jack -neon -nsplugin% (-pgo) (-pulseaudio) (-selinux) (-system-cairo) -system-sqlite {-test} -wifi $ -as -ast -az -bg -bn-BD -bn-IN -br -bs -ca -cak -cs -cy -da -de -dsb -el -en-ZA -eo -es-AR -es-CL -es-ES -es-MX -et -eu -fa -ff -fi -fr -fy -ga -gd -gl -gn $ -km -kn -ko -lij -lt -lv -mai -mk -ml -mr -ms -nb -nl -nn -or -pa -pl -pt-BR -pt-PT -rm -ro -ru -si -sk -sl -son -sq -sr -sv -ta -te -th -tr -uk -uz -vi -xh$

mount tmpfs -t tmpfs -o rw,nosuid,nodev,noexec /dev/shm

Errors like File "/usr/lib64/python2.7/multiprocessing/synchronize.py", line 75, in __init__ sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue) OSError: [Errno 38] Function not implemented indicate that /dev/shm is not available in the chroot.