Raspberry Pi 3 64 bit Install

History
Its been about a year since the Raspberry Pi3 with a 64bit capable CPU became available. At the outset, it was difficult to install Gentoo on the Pi 3 in 64 bit mode. A lot of work by a lot of people has almost brought a 64 bit Gentoo install on the Pi 3 down to almost a standard handbook install.

The Gentoo arm64 offering is experimental at the time of writing. That means that very little is marked stable. Expect to use ACCEPT_KEYWORDS="~arm64" and then use package.accept_keywords too.

What Works
All the Pi 3 hardware is supported in 64 bit mode. That's WiFi USB Ethernet (needs USB) Bluetooth Hardware Video Acceleration Sound over HDMI

Untested
PAL/NTSC Video Output Analogue Sound Output

Whats Required
Gentoo Install on a PC microSD card reader for the PC Raspberry Pi 3 microSD card > 8G USB Keyboard USB Mouse HDMI Display The content of the microSD card will be wiped during the install.

Installation Overview

 * Install crossdev on the PC
 * Fetch the Raspberry Pi firmware
 * Fetch the Raspberry Pi kernel
 * Partition the microSD card
 * Fetch the Gentoo bits of the install
 * Cross compile and install your kernel
 * Setup
 * Boot the Pi to Test

Install Crossdev on the PC
There are no prebuilt kernel images for the Pi3 in its arm64 mode. Until you have a arm64 kernel, you can't boot it in 64 bit mode. Its a little chicken and egg. Once the Pi is running, it can build its own kernels.

crossdev is Gentoos' tool for building cross compiler tool chains. Once its installed, we will use it to build the arm64 kernel on the Gentoo PC. will install the crossdev tool.

Using Crossdev to Build A Cross Compiler
You may need to nominate an overlay to store the cross ebuilds, crossdev will let you know.

There are other parameters you can pass to crossdev too.

Convert files as crossdev asks e.g. error: please convert /etc/portage/package.env to a directory by appending _file to the existing filename

making the directory

then moving package.env_file into the directory.

Rinse and repeat until crossdev is happy.

crossdev will take a while. Its building binutils:             binutils-[latest] gcc:                  gcc-[latest] headers:              linux-headers-[latest] libc:                 glibc-[latest]

When crossdev completes you will have a cross toolchain

It will also create an arm64 target root in /usr/aarch64-unknown-linux-gnu/ This is used by cross emerge.

Pure cross compiling, other than the kernel, is out of scope of this guide.

Fetch the Raspberry Pi Firmware
The Raspberry Pi Firmware is maintained in a git repository. You will need to install git if you don't have it.

Don't do anything you don't need to do as root.

As your normal user, make some space in the home directory for raspberry pi will do nicely.

This will be for the Raspberry Pi Firmware and kernel

will fetch the Pi firmware into a directory called firmware

It will actually fetch all the history too. If you are short of space or bandwidth, a shallow clone is all that's needed. A full clone is 6.3G and growing, we only need the /boot directory out of the master branch.

There is nothing to build. ~/raspberrypi/firmware/boot is used as is.

Fetch the Raspberry Pi Kernel
Stay in raspberrypi and

will fetch the kernel into a directory called linux.

With absolutely no fanfare at all, 64 bit support was added to this kernel tree late in 2016. No more searching for odd patches.

Not everything has been accepted by the mainline kernel yet but its getting closer. Feel free to test for yourself.

The master branch may be broken as its commit by commit as it happens. Test if you want to. Its better to use an identified tag. At the time of writing that is rpi-4.10.y. The 4.10 mainline kernel is still at -rc status.

That's the kernel source tree in place, ready for configuring and cross compiling.

Configure The Kernel
As your user

The bcmrpi3_defconfig is almost right as it stands. It defaults to the powersave CPU governor, which runs the Pi at 600MHz. All the governors are there. Ondemand is recommended.

Either use menuconfig to change the default CPU governor, add something to the kernel command line, or change it after booting.

Search for CPU_FREQ_DEFAULT_GOV go to that location and set the default to Ondemand. Exit menuconfig, saving the change.

.config - Linux/arm64 4.10.0-rc6 Kernel Configuration > CPU Power Management > CPU Frequency scaling ────────────────────────────────── ┌────────────────────────── CPU Frequency scaling ───────────────────────────┐ │ Arrow keys navigate the menu.  selects submenus ---> (or empty   │    │  submenus ). Highlighted letters are hotkeys. Pressing  includes, │ │  excludes,  modularizes features. Press  to exit, <?>   │ │ for Help,  for Search. Legend: [*] built-in [ ] excluded   module │ │ ┌────────────────────────────────────────────────────────────────────────┐ │   │ │    [*] CPU Frequency scaling                                           │ │ │ │   [*]   CPU frequency transition statistics                           │ │ │ │   [ ]     CPU frequency transition statistics details                 │ │ │ │         Default CPUFreq governor (powersave)  --->                    │ │ │ │   <*>   'performance' governor                                        │ │ │ │   -*-   'powersave' governor                                          │ │ │ │   <*>   'userspace' governor for userspace frequency scaling          │ │ │ │   <*>   'ondemand' cpufreq policy governor                            │ │

The kernel .config contains lots of support for hardware you don't have and possibly have never heard of. More confident readers may be tempted to trim things out now. A word of advice - don't, at least, not until the system boots.

Cross Compiling The Kernel
The kernel will not link with the gold linker. It ends with aarch64-unknown-linux-gnu-ld: fatal error: -shared and -pie are incompatible If you don't know what the gold linker is, you are not using it.

The build is conventional, other than telling the build system to build for arm64 and use the cross compiler.

Change X to the number of parallel MAKE jobs you want to run. Convention is cores+1.

Background
The Pi does not need a boot loader. There is no firmware for the arm CPU, like there is in a PC. Instead, the GPU manages loading software for the CPU to execute, while the CPU is held reset. Its the GPU that has the 'BIOS' that gets everything started.

This makes it impossible to brick the Pi, since at worst, the microSD card needs to be reloaded.

This arrangement does impose some constraints on the microSD card. The partition table must be MSDOS The first partition (/boot) must be vfat.

The bootcode.bin file has some useful defaults that make setup easier, which we will take advantage of later.

Partitioning
Depending on how the microSD card is connected to you PC, it my be /dev/sdX or /dev/mmcblkY

In the example below, its /dev/sdk. Using the partitioning tool of your choice, make three partitions on your microSD card. boot 128Mb swap 2G root (/) the rest.

Using fdisk, and your microSD card block device, not my /dev/sdk

Make The Partition Table and Add Partitions
Make a new MSDOS disklable Command (m for help): o

Add a new partition - this will be 128Mb for /boot Command (m for help): n Partition type p  primary (0 primary, 0 extended, 4 free) e  extended (container for logical partitions) Select (default p): p Partition number (1-4, default 1): 1 First sector (2048-15523839, default 2048): Last sector, +sectors or +size{K,M,G,T,P} (2048-15523839, default 15523839): +128M Has created a new partition 1 of type 'Linux' and of size 128 MiB.

Add a 2G partition 2 for swap.

Put the remaining space in partition 3 for root. Check with Command (m for help): p Using an 8G microSD card it should show Device    Boot   Start      End  Sectors  Size Id Type /dev/sdk1         2048   264191   262144  128M 83 Linux /dev/sdk2       264192  4458495  4194304    2G 83 Linux /dev/sdk3      4458496 15523839 11065344  5.3G 83 Linux

Set Flags and Partition Types
Toggle the bootable flag on partition 1 Command (m for help): a Partition number (1-3, default 3): 1 The bootable flag on partition 1 is enabled now.

Mark partition 1 as FAT Command (m for help): t Partition number (1-3, default 3): 1 Partition type (type L to list all types): c

Changed type of partition 'Linux' to 'W95 FAT32 (LBA)'.

Mark partition 2 as swap Command (m for help): t Partition number (1-3, default 3): 2 Partition type (type L to list all types): 82

Changed type of partition 'Linux swap / Solaris' to 'Linux swap / Solaris'.

Check again. Device    Boot   Start      End  Sectors  Size Id Type /dev/sdk1 *       2048   264191   262144  128M  c W95 FAT32 (LBA) /dev/sdk2       264192  4458495  4194304    2G 82 Linux swap / Solaris /dev/sdk3      4458496 15523839 11065344  5.3G 83 Linux

Save The New Partiton Table
Exit fdisk with either w or q  w   write table to disk and exit q  quit without saving changes

Make Filesystems
You know the block devices from the partitioning step above. Fill in the ... to match your system.

vfat for /boot

mkswap for swap

ext4 for root

The i-node count cannot be changed after filesystem creation and can limit the number of files on a files system. The Gentoo repository alone needs over 17,000 i-nodes.

Fetch the Gentoo bits of the install
To make it easy to cross refer to the Gentoo_Handbook

Mount the microSD card root filesystem at /mnt/gentoo

Install the arm64 Stage 3
Following the Gentoo_Handbook fetch the arm64 stage3 and untar it to /mnt/gentoo in the normal way.

/mnt/gentoo/tmp should be empty. Clear it now or the the Pi will do it at boot. That may take a long time.

Before and after clearing /mnt/gentoo/tmp Filesystem               1K-blocks      Used Available Use% Mounted on /dev/sdk3                   5380212   1205968   3881228  24% /mnt/gentoo /dev/sdk3                  5380212    874636   4212560  18% /mnt/gentoo

Install a Portage Snapshot
This step is not actually needed to boot the Pi but emerge won't work without it

Following the Gentoo_Handbook fetch and unpack a portage shapshot in the normal way.

Careful readers can copy their host /usr/portage as long as ./packages and ./distfiles are omitted.

The Raspberry Pi Foundation Provided Files
Mount the microSD card boot at /mnt/gentoo/boot. (The microSD card root should still be mounted at /mnt/gentoo)

As root, copy the content of your normal users ~/raspberrypi/firmware/boot to /mnt/gentoo/boot.

This will generate a warning cp: failed to preserve ownership for ' ': Operation not permitted for every file. vfat does not support permissions, so the warning are expected and can be ignored.

You should end up with files in /mnt/gentoo/boot, not a directory called boot.

Not all the files there are required. Its a one size fits all for all models of Raspberry Pi operating in 32 bit mode.

Install the Kernel to the microSD Card
The kernel was built above, now to install it.

The kernel is in three parts kernel binary kernel modules the device tree

Install The Kernel Binary
As root, copy the kernel binary from the build location

It is possible to use other kernel file names by adding entries to /boot/config.txt

Install The Device Tree
The device tree binary (.dtb) describes the hardware to the kernel. This avoids having all the existing hardware configurations ard coded into the kernel.

Due to the way the Raspberry Pi 64 bit kernel support has been added, there are going to be two different device trees with the same file name. A 32 bit version and a 64 bit version. They are not interchangeable. Move the 32 bit version out of the way. That's renamed the 32 bit version.

Copy the dtb from the build location

It is possible to use other device tree file names by adding entries to /boot/config.txt

Install The Kernel Modules
From the top of the kernel tree, install the kernel modules.

The INSTALL_MOD_PATH is the root of the filesystem the modules are to be installed into. Due to kernel naming its unlikely you will have a 4.10.0-rc6-v8+ so omitting INSTALL_MOD_PATH is probably harmless to your build host install.

Checking The Kernel Install
COPYING.linux          bcm2708-rpi-cm.dtb      bcm2710-rpi-cm3.dtb  fixup_db.dat  kernel8.img   start_db.elf LICENCE.broadcom       bcm2709-rpi-2-b.dtb     bootcode.bin         fixup_x.dat   overlays      start_x.elf bcm2708-rpi-b-plus.dtb bcm2710-rpi-3-b.dtb     fixup.dat            kernel.img    start.elf bcm2708-rpi-b.dtb      bcm2710-rpi-3-b.dtb_32  fixup_cd.dat         kernel7.img   start_cd.elf

Notice the two bcm2710-rpi-3-b.dtb device tree binaries and the kernel8.img.

4.10.0-rc6-v8+ Shows that the kernel modules were installed to the correct location.

Raspberry Pi 3 Peripherals
Now that we have the base operating system in, we will need to do some file configuration by hand to get the peripherals working.

Serial Port Configuration
The Gentoo Stage3 comes with the default Gentoo serial port configuration. In Raspbian Jessie, Udev rules exist that provide aliases for GPIO14:15 (/dev/serial0) and the Bluetooth serial port (/dev/serial1). If you're not using the pi3-miniuart-bt or the pi3-disable-bt device tree overlays, ttyS0 (mini UART) points to /dev/serial0 (GPIO14:15) while ttyAMA0 (Bluetooth) points to /dev/serial1 (Bluetooth).

If, however, the pi3-miniuart-bt overlay is in use, Udev rules automagically make ttyAMA0 point to /dev/serial0 (GPIO14:15) while ttyS0 points to /dev/serial1 (Bluetooth). If the pi3-disable-bt overlay is in use, ttyAMA0 still points to /dev/serial0 (GPIO14:15) while Bluetooth/ttyS0 are disabled.

This allows the use of /dev/serial1 in any Bluetooth configuration files and command line arguments regardless of which serial port it is assigned to. It also allows the use of /dev/serial0 in any apps and/or configuration files/command line arguments which reference the GPIO14:15 pins regardless of which overlay is/is not in use.

In order to make things easier when working with the device tree overlays for the Bluetooth module as well as any applications which were written in a Raspbian environment that uses the serial0 and serial1 aliases, we'll need to first create a Udev rule to mimick this behavior.

Open up /etc/inittab -

Find this line and comment it out by appending a # at the beginning of the line -

Save and exit the file. This prevents Gentoo from assigning the serial console to ttyAMA0 on /root mount, which will conflict with Bluetooth operation if left uncommented.

Next, create the empty file 99-com.rules in /etc/udev/rules.d -

Copy and paste this into the empty file, then save and exit the file -

This will assign ttyS0 and ttyAMA0 to the dialout group just as they are in Raspbian Jessie. It will also provide the serial0 (GPIO14:15) and serial1 (Bluetooth) aliases, which eases the task of switching the serial ports around between the Bluetooth and GPIO14:15.

Install WiFi Firmware
The on board wifi needs firmware. The files brcmfmac43430-sdio.txt and brcmfmac43430-sdio.bin are both required to be present in /mnt/gentoo/lib/firmware/brcm/

They can be downloaded via wireless.wiki.kernel.org. Alternatively, you can also pull them from /lib/firmware/brcm on an existing Raspbian installation. Copy the entire /lib/firmware/brcm directory from a Raspbian install into the Gentoo /lib/firmware directory.

Kernel
You'll also want to ensure that Broadcom IEEE802.11n embedded FullMAC WLAN driver with SDIO bus interface support is enabled in the kernel. In the menu shown below, this is all that needs to be enabled. You can set the FullMAC WLAN driver as a module or compile it into the kernel. If you plan to use only the on board WiFi, everything else under the Wireless LAN menu can be disabled.

Once the firmware files are copied over to /lib/firmware/brcm, the appropriate kernel settings are built, and a wireless network manager (such as wpa_supplicant) is installed, the WiFi should work.

Install Bluetooth Firmware
The on board bluetooth needs the firmware files bcm43xx-0.fw and bcm43xx_hdr-0.fw in /mnt/gentoo/lib/firmware.

You will also need to install the Bluez package, which is the Linux Bluetooth stack. It also includes the necessary commands hciattach, which is used by the startup script below to attach the Bluetooth module to a UART, and hciconfig. It must be built with the depricated USE flag enabled -

Or you can simply do -

Also required is the bluez-tools package. This provides the hciattach command, which will allow your system to attach the Bluetooth module to the UART, as shown in the following startup script. ** will need to be added to package.accept_keywords in order to emerge it -

It also needs a startup script in /mnt/gentoo/etc/local.d/bluetooth.start

/usr/bin/hciattach /dev/ttyAMA0 bcm43xx 921600 noflow -
 * 1) !/bin/bash
 * 2) Bluetooth is attached to /dev/ttyAMA0 so attach it and
 * 3) load the firmware

Setup
The chroot environment is very handy as it allows you to configure your startup run levels (rc-update) as well as update your Portage snapshot (emerge --sync). Additionally, you can set the root password as well as create users in the chroot environment.

If you plan to only access your Pi via SSH, you'll need at least networking on startup, which requires either net.eth0 or dhcpcd to be added to the default run level, as well as OpenSSH itself (sshd).

Much of the setup prior to booting a new Gentoo install is done in a chroot. Chrooting into an arm64 install from some other arch is beyond the scope of this guide. Instead, only the bare minimum setup from outside the chroot is covered. Consult the Embedded Handbook/General/Compiling with qemu user chroot page to learn how to use Qemu to establish a cross-chroot.

Root Password
There are several ways to generate a password hash for /etc/shadow I usually copy the hash from another system. My Pi uses an /etc/shadow root entry root:$6$xxPVR/Td5iP$/7Asdgq0ux2sgNkklnndcG4g3493kUYfrrdenBXjxBxEsoLneJpDAwOyX/kkpFB4pU5dlhHEyN0SK4eh/WpmO0::0:99999:7::: The matching password is raspberry.

Feel free to use that line to replace the root entry in /mnt/gentoo/etc/shadow

You can change the password once you are logged in.

/etc/fstab
On the Pi, the microSD card will be /dev/mmcblk0 with partitions /dev/mmcblk0p1, /dev/mmcblk0p2 and /dev/mmcblk0p3

Edit /mnt/gentoo/etc/fstab to match.

/dev/mmcblk0p1         /boot           vfat            noauto,noatime  1 2 /dev/mmcblk0p2         none            swap            sw              0 0 /dev/mmcblk0p3         /               ext4            noatime         0 1

/boot/config.txt
This file will not exist until its created. Create it with the following content.

Alternative: Use the /boot/config.txt From A Working Raspian Install
Here is a config.txt file from a working Raspian install. If the above does not work, this one should.

/boot/cmdline.txt
This file will not exist until its created. Create it with the following content.

This boots the Pi with a kernel command line of 8250.nr_uarts=0 cma=256M@256M dma.dmachans=0x7f35 bcm2708_fb.fbwidth=1920 bcm2708_fb.fbheight=1080 bcm2709.boardrev=0xa02082 bcm2709.serial=0x8e2830fe smsc95xx.macaddr=B8:27:EB:28:30:FE bcm2708_fb.fbswap=1 bcm2709.uart_clock=48000000 vc_mem.mem_base=0x3dc00000 vc_mem.mem_size=0x3f000000 root=/dev/mmcblk0p3 rootfstype=ext4 rootwait

It really is one long line.

Alternative: Use /boot/cmdline.txt From A Working Raspian Install
Here is the cmdline.txt file from a working Raspian install. If the above does not work, this one should. Be sure and edit the root= directory to your root partition and the rootfstype= to your partition type. /dev/mmcblk0p7 and ext4 are shown below as examples.

Setting the Console Keymap
This step is optional if you can log in using the default keymap="us"

Set the keymap to something you use, e.g. keymap="dvorak-uk"

Boot the Pi to Test
Unmount the microSD card.

When the prompt returns, move the microSD card to the Raspberry Pi and power on.

For 10 seconds (it seems much longer) you should see the GPU 'Rainbow' test pattern, then the familiar boot messages.

Log in at the Pi console. Nothing was added to any runlevels during the install, so networking was not started, nor anything that depends on networking, like ntpd and sshd.

The Pi does not have a hardware real time clock. Its time will be Jan 1, 1970.

What Next
As always with Gentoo, if it booted, that's the hard bit done.

All The setup steps in the Gentoo Handbook Fix the MAC address or use a static IP Allow root logins via ssh Add a crond, a logger and other things the handbook does before the reboot. Add Kernel Sources (or at least the .config)

WiFi and Bluetooth
Its unlikely that WiFi or Bluetooth will work at first boot. Expect to add some control tools for both.

CFLAGS
CFLAGS="-march=armv8-a+crc -mtune=cortex-a53 -ftree-vectorize -O2 -pipe -fomit-frame-pointer"

gcc-6.x allows the use of -march=native but that will prevent the use of distcc. The above is the same as gcc-6.3 would set for -march=native anyway.

ACCEPT_KEYWORDS
Outside of the @system set, arm64 is either testing or keyword masked. Set ACCEPT_KEYWORDS="~arm64" and expect to use package.accept_keywords too.

Upgrade gcc then rebuild all of the installed C++ software.

MAKEOPTS
With only 1G RAM, and four cores, the conventional MAKEOPTS="-j5" is a bit aggressive for building larger things. It will force swapping or even appear to lock up the Pi completely, to the point where it won't even respond to the console.

Use files in /etc/portage/env/ and entries in /etc/portage/package.env to set MAKEOPTS on a per package basis.

Networking
dhcpcd is not in the stage 3, nor is eth0 in the default runlevel. will bring up eth0

sshd
The default configuration for sshd will not allow password based root logins. add your ssh public key for root make a normal user in the wheel group edit /etc/ssh/sshd_config to allow password based root logins

Updating The Tool Chain
Once you boot, you may have the desire to update @world first thing. However, as of the time of this writing, the latest stage3 for arm64 was built in December 2016. A lot of things in the tool chain will be out of date with what is on the current Portage tree. Once you've booted the Raspberry Pi and confirmed that you have an internet connection, you'll want to first run emerge --sync to get the absolute latest tree, then run perl-cleaner --all to get all of your Perl packages up to date.

Network Time Sync
The Raspberry Pi 3 does not have a hardware real time clock on board. There are vendors online where you can order RTC modules made for the Pi, but if you don't plan to run one, I highly recommend installing a NTP client.

First, we'll set the initial time using the 'date' command. Date and time will be entered in mmddhhmmyyyy format and the time is in 24-hr format -

As an example, if the time is 10:05PM on 7/31/2017 -

As with most things Gentoo, the NTP daemon is just an emerge away -

Remove the hardware clock service hwclock from the boot runlevel and replace it with the software clock service swclock -

Make sure you have the correct time zone set to the area which most closely matches your locale in /usr/share/zoneinfo -

As an example, if you live in California, you would do -

Install your timezone libraries -

Start the NTP client and add it to the default runlevel -

Where to Get Help
On Internet Relay Chat irc.freenode.net#gentoo-arm irc.freenode.net#gentoo-embedded

On the Gentoo Forums, start a new topic in the Gentoo on ARM forum.

I don't mind a PM on the forums with a link to your post. I don't do one to one help via email or the forums PM system. You will either get no response at all or a request to make a public post. That way others may learn from your misfortune.

Acknowledgements
Everyone contributing to the arm64 software base.

Especially Sakaki, who showed the way on the final steps.