User:NeddySeagoon/Pi3

Neddys Notes to self to get Gentoo on a Raspberry Pi 3 in 64 bit mode
This page is hopelessly out of date. It can be regarded as the 'hair shirt' method of running 64 bit Gentoo on a Raspberry Pi 3.

The cross compile bits are still valid but there is a DAS-U-Boot, a kernel tree and .dts that just works, so getting /boot right is not nearly as difficult as described here.

Skip to the end of the document if you just want to plug and play.

Background
Its been 4 years since the Raspberry Pi B was launched' with 256Mb RAM and a single core, 700MHz, armv6j CPU.

Installing Gentoo was pretty much a matter of following the handbook and adding in cross distcc to speed things up.

The Pi3 is provided with 32 bit software. Its a four core armv7a CPU on the surface. The tried and tested install methods should still work.

If you just want to drive your Ferrari to the corner shop, that's well documented on this wiki

The article is about running Gentoo on the Pi 3 in 64 bit mode, as an armv8a. That's what the CPU is capable of. There is no handbook nor any stage files for that yet.

The Raspberry Pi 3 is a bit of a dog on the I/O front. The four USB ports and the Ethernet port are all on a singe USB2 interface from the CPU. That's unchanged is the last four years. That gives 400Bit/sec for everything connected there,

The new WiFi and Bluetooth are grafted onto the SD card interface, so they contend for the available bandwidth too.

Overview
As there is no stage files, the conventional chroot install is out of the question. There is nothing to chroot into. The workaround for that is to build some armv8a code on some other system in a process known as cross compiling.

Cross compiling is the process of building packages on one architecture, (the build host) that are intended to run an another architecture (the target). I will be building on amd64 for armv8a.

That's a little bit of an exaggeration. If your 64 bit cross compiled kernel can get as far as a console, you may get kernel error messages.

Serial Console
A 3.3v serial port adaptor is required. You will need a USB to serial port adaptor as PC serial ports if you have one, uses either +/- 5v or +/-12v, depending on the age of your PC. Either will destroy a Pi serial port and may be more.

Das U-Boot
The Raspberry Pi does not normally use a boot loader. The boot files are loaded by the firmware. However, the CPU needs to be at the correct privilege level when control is passed to the kernel, or the kernel won't run. Das U-Boot is loaded in the normal way. It then loads the kernel and passes control to it with the correct privilege level for the kernel to run.

We need a 64 bit arm8a Das U-Boot to get things started.

Kernel
To run 64 bit code we need a 64 bit armv8a kernel. Once the CPU is switched into 64 bit mode, before even Das U-Boot is loaded, the 32 bit kernel will not run. Even if it did, it would not run 64 bit code.

Cross Compiled Userspace
Once the kernel runs successfully, it will try to run /sbin/init, or whatever init= points to.

Systemd users are on there own. This author does not use systemd and is not about to start while bringing up 64 bit Gentoo on a Raspberry Pi3.

We need a cross tool chain and a target root filesystem to put all the armv8a code into. Its a really bad idea to emerge armv8a code to your build host.

Getting Started
On your main Gentoo system. There are other parameters you can pass to crossdev too.

This will create a cross toolchain that builds things into /usr/aarch64-unknown-linux-gnu/ This is the target root.

Eventually, you copy this directory to / on your Pi SDCard.

Once crossdev has finished, you will have a /usr/aarch64-unknown-linux-gnu/etc/portage which corresponds to your build host /etc/portage. The difference being that /usr/aarch64-unknown-linux-gnu/etc/portage applies when cross compiling for aarch64-unknown-linux-gnu.

Any number of cross compilers con be installed. Practice on something more well proven before you try 64 bit Raspberry Pi.

Target configuration
This consists of setting up /usr/aarch64-armv8a-unknown-gnu/etc/portage by hand.

make.conf
Copy your mirror list from the build host and any of your favourite options.

Optimists can set VIDEO_CARDS= and so on. Just like the Gentoo on your build host, these settings can be changed as needed.

Note that the default settings will share the hosts portage tree and distfiles but keep binary packaces and ongoing builds separate from the build hosts. This means its permissible to build for several targets at the same time.

cross make.profile
The default generic make.profile symbolic link needs to be changed. The cross make.profile (by hand)

Or use another arm64 profile that's more to your liking. Don't use eselect profile.

Cross Compiling the Kernel
Mainline kernels do not have the Raspberry Pi3 64 bit patches. Fetch a kernel using git.

Make the normal home for the kernel.

With thanks to marcus_c at the Raspberry Pi Forum. https://www.raspberrypi.org/forums/viewtopic.php?f=72&t=137963&start=25

Cross Compiling Das U-Boot
May as well put Das U-Boot beside the kernel,

With thanks to marcus_c at the Raspberry Pi Forum. https://www.raspberrypi.org/forums/viewtopic.php?f=72&t=137963

Cross Building User Space Packages
Execute

This sets up some symlinks so that

uses emerge, with the cross toolchain to install armv8a packages into /usr/aarch64-unknown-linux-gnu

emerge-wrapper needs to run every time you open a new shell to cross compile in.

Random Warnings And Caveats
/usr/aarch64-unknown-linux-gnu/ is empty when you start to cross compile, unlike a stage3 install, where a working host chroot is provided. None of the code here will run on the build host.

In theory will cross compile the system set. It will have a good go but it will fail for a number of reasons. * To avoid circular dependencies the @system set is assumed to always be present. * Some build systems don't support cross compiling * arm64 is not well tested - don't even think of using ACCEPT_KEYWORDS=arm64 * expect to use the keyword ** on a number of packages

USE=-cxx aarch64-unknown-linux-gnueabi-emerge @system -uDNav almost works.

Missing deps - fix by hand

gcc - uses wrong linker /usr/lib/gcc/x86_64-pc-linux-gnu/5.3.0/../../../../x86_64-pc-linux-gnu/bin/ld

sys-apps/util-linux fails configure: error: udev selected but libudev not found even when libudev built

sys-devel/llvm needs dev-util/ninja. dev-util/ninja builds then sys-devel/llvm fails later when the build system tries to execute armv8a code on the build host.

groff

checking whether g++ accepts -g... yes checking that header files support C++... no configure: error: header files do not support C++

udev builds eudev fails.

armv8a is not well tested - be prepared keyword things you want to try

Optional Additions to @system USE="-cxx -fortran" aarch64-armv8a-linux-gnueabi-emerge dcron metalog ntpd dhcpcd wpa_supplicant

libepoxy

graphite2 -- Check for working CXX compiler: /usr/bin/g++ -- Check for working CXX compiler: /usr/bin/g++ -- broken CMake Error at /usr/share/cmake/Modules/CMakeTestCXXCompiler.cmake:54 (message): The C++ compiler "/usr/bin/g++" is not able to compile a simple test program.

libical -- Check for working CXX compiler: /usr/bin/g++ -- Check for working CXX compiler: /usr/bin/g++ -- broken CMake Error at /usr/share/cmake/Modules/CMakeTestCXXCompiler.cmake:54 (message): The C++ compiler "/usr/bin/g++" is not able to compile a simple test program.

fortune-mod /bin/sh: line 6: ../util/strfile: cannot execute binary file: Exec format error

libgpg

gnome-icon-theme checking icon-naming-utils >= 0.8.7... no configure: error: icon-naming-utils >= 0.8.7 is required to build and install gnome-icon-theme That needs perl, which is designed to not cross compile.

mupdf

dialog /usr/lib/gcc/x86_64-pc-linux-gnu/5.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: .libs/trace.o: Relocations in generic ELF (EM: 183) .libs/trace.o: error adding symbols: File in wrong format collect2: error: ld returned 1 exit status

screen ... configure: error: in `/usr/aarch64-armv8a-linux-gnueabi/tmp/portage/app-misc/screen-4.3.1-r1/work/screen-4.3.1': configure: error: cannot run test program while cross compiling See `config.log' for more details dbus-glib

Missing Dirs
The /usr/aarch64-unknown-linux-gnu/ directory has a few key directories missing that are needed to boot. A few are nice to have.

Set the owner, group and modes to match the build host.

/etc/fstab
Only the SDCard slot and the serial console will work. Boot must be on the SDCard and in must be in vfat format.

Root can be anywhere but the SDcard is a good choice right now.

This /etc/fstab is consistent with the rest of this page but this being Gentoo you are free to change it to suit yourself.

Beware of using partitions beyond /dev/mmcblk0p7 as the kernel defaults to eight minor device numbers for mmc cards. Thats mmcblk0 and mmcblk0p1 ... mmcblk0p7.

/dev/mmcblk0p1		/boot		vfat		noauto,noatime	1 2 /dev/mmcblk0p2		/		ext4		noatime		0 1
 * 1) 			 	 		 		
 * 1) NOTE: If your BOOT partition is ReiserFS, add the notail option to opts.
 * 1) None of these ever
 * 2) /dev/cdrom		/mnt/cdrom	auto		noauto,ro	0 0
 * 3) /dev/fd0		/mnt/floppy	auto		noauto		0 0

Add a swap later.

mtab
Make /etc/mtab -> /proc/self/mounts in /usr/aarch64-unknown-linux-gnu/

/etc/inittab
A serial console must be configured. Das U-Boot can drive the HDMI output but the kernel cannot.

The last message that appears on the HDMI is "Starting Kernel ..."

Edit /etc/inittab for the first serial terminal. s0:12345:respawn:/sbin/agetty -L 115200 ttyAMA0 vt340
 * 1) SERIAL CONSOLES

and comment out the 9600 baud entry below. #f0:12345:respawn:/sbin/agetty 9600 ttyAMA0 vt100 This avoids the serial terminal baud rate or the Pi being changed to 9600 baud, while the terminal emulator works at 115200 baud.
 * 1) Architecture specific features

Clear /usr/aarch64-unknown-linux-gnu/tmp
This is the portage build location for aarch64-unknown-linux-gnu cross compiles. On the SDcard, it will become /tmp, which gets cleared every boot. There will be a lot of junk there, so you may need to wait a long time for the Pi to clear it if you forget.

Copy /usr/aarch64-unknown-linux-gnu/ to the root partition on your microSD card. This is in place of what was there.

Copy u-boot.bin, the dtb and kernel to the boot partition on your microSD Card. This is in addition to the files already there.

config.txt
There are few changes to config.txt but back it up anyway. Add the following ... arm_control=0x200 kernel=u-boot.bin dtoverlay=pi3-miniuart-bt-overlay kernel_old=1 disable_commandline_tags=1
 * 1) boot in 64 bit mode !!!

force_turbo=1
 * 1) get the serial console baud rate right

arm_control=0x200 sets the CPU into 64 bit mode.

kernel=u-boot.bin starts u-boot instead of a kernel.

dtoverlay=pi3-miniuart-bt-overlay gets the details for the serial port on the Pi.

disable_commandline_tags disables any kernel command line built into the kernel.

force_turbo=1 is needed to stop the serial port baud rate changing with CPU clock speed.

Configuring the system
Follow Configuring the system in the handbook. Adjust the paths given. Root is now.

Booting In 64 Bit Mode
Best test your serial console in 32 bit mode first. Its all you will have so you want to know it works

First Boot Only
To automatically boot with Das U-Boot, it reads its u-boot.env to know what to do. It looks like it may be a text file but its not quite.

At the U-Boot prompt, enter the following, a line at a time.

setenv linux_load 'fatload mmc 0:1 $loadaddr image' setenv fdt_load 'fatload mmc 0:1 $fdt_addr_r $fdtfile' setenv fdtfile 'bcm2837-rpi-3-b.dtb' setenv bootcmd 'run fdt_load ; run linux_load ; booti $loadaddr - $fdt_addr_r' setenv bootargs 'dwc_otg.lpm_enable=0 console=ttyAMA0,115200 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait' saveenv

This creates your u-boot.env

Just like more common boot loaders, bootargs must be a single line. From time to time, its useful to add init=/bin/bash to bookargs.

Power cycle the Pi or type boot to test.

What could Possibly Go Wrong?
Booting stops at * Wiping /tmp directory ... All the build leftovers are still there. This is where the cross compiler puts its portage build directory. Make a coffee, it will eventually clear /tmp and move on.

The serial console stops at * Starting local ... [ ok ]

/etc/inittab is in a mess. You should see * Starting local ... [ ok ] This is localhost.unknown_domain (Linux aarch64 4.5.0+) 00:00:10 localhost login:

Both ends of the serial link are running with different setting. Lucky users will get gibberish, so its obvious that there are serial link issues.

The Easy Way
Practice with your 3.3v serial console

Take an 8G (or larger) microSD card. Make a DOS disklabel on it. Use fdisk to make two partitons

Partition 1 needs to be 64MB, formatted as vfat. This will be /boot

Partition 2 needs to be 5GB, formatted as ext4. This will be / (root)

Fetch http://grytpype-thynne.org/~roy/Pi3_64bit/Pi3_64_boot.tar.xz and untar it to partition 1

Fetch http://grytpype-thynne.org/~roy/Pi3_64bit/Pi3_64_root.tar.xz and untar it to partition 2

Move the microSD card to the Pi3 and power up.

The only user is root and the password is raspberry.

Ignore the can't mount /usr/portage and /usr/portage/distfiles. They are listed in /etc/fstab along with swap but are not provided in the image.