Raspberry Pi 3 64 bit Install

From Gentoo Wiki
Jump to: navigation, search

Overview

Important
This Page Has Been Tested Once and found to WFM. It Might Still Drink All The Beer in Your Fridge ... or Worse

History

Its been about a year since the Raspberry Pi3 with a 64bit capable CPU became available. At the outset, it was difficult to install Gentoo on the Pi 3 in 64 bit mode. A lot of work by a lot of people has almost brought a 64 bit Gentoo install on the Pi 3 down to almost a standard handbook install.

The Gentoo arm64 offering is experimental at the time of writing. That means that very little is marked stable. Expect to use ACCEPT_KEYWORDS="~arm64" and then use package.accept_keywords too.

What Works

All the Pi 3 hardware is supported in 64 bit mode. That's

WiFi
USB
Ethernet (needs USB)
Bluetooth
Hardware Video Acceleration
Sound over HDMI

Untested

PAL/NTSC Video Output
Analogue Sound Output

Whats Required

Gentoo Install on a PC 
microSD card reader for the PC
Raspberry Pi 3
microSD card > 8G
USB Keyboard 
USB Mouse
HDMI Display 

The content of the microSD card will be wiped during the install.

Installation Overview

  • Install crossdev on the PC
  • Fetch the Raspberry Pi firmware
  • Fetch the Raspberry Pi kernel
  • Partition the microSD card
  • Fetch the Gentoo bits of the install
  • Cross compile and install your kernel
  • Setup
  • Boot the Pi to Test

Install Crossdev on the PC

There are no prebuilt kernel images for the Pi3 in its arm64 mode. Until you have a arm64 kernel, you can't boot it in 64 bit mode. Its a little chicken and egg. Once the Pi is running, it can build its own kernels.

crossdev is Gentoos' tool for building cross compiler tool chains. Once its installed, we will use it to build the arm64 kernel on the Gentoo PC.

root #emerge --ask crossdev
will install the crossdev tool.

Using Crossdev to Build A Cross Compiler

root #crossdev -t aarch64-unknown-linux-gnu

You may need to nominate an overlay to store the cross ebuilds, crossdev will let you know.

There are other parameters you can pass to crossdev too.

Important
Do not use -S to build a stable toolchain, arm64 moves quickly
Important
Crossdev insists that many of the files in /etc/portage/ are directories

Convert files as crossdev asks e.g.

error: please convert /etc/portage/package.env to a directory

by appending _file to the existing filename

root #mv /etc/portage/package.env /etc/portage/package.env_file

making the directory

root #mkdir /etc/portage/package.env

then moving package.env_file into the directory.

root #mv /etc/portage/package.env_file /etc/portage/package.env

Rinse and repeat until crossdev is happy.

crossdev will take a while. Its building

binutils:              binutils-[latest]
gcc:                   gcc-[latest]
headers:               linux-headers-[latest]
libc:                  glibc-[latest]

When crossdev completes you will have a cross toolchain

user $$ gcc-config -l
[1] aarch64-unknown-linux-gnu-5.4.0 *
[2] x86_64-pc-linux-gnu-5.3.0
[3] x86_64-pc-linux-gnu-5.4.0 *

It will also create an arm64 target root in /usr/aarch64-unknown-linux-gnu/ This is used by cross emerge.

Pure cross compiling, other than the kernel, is out of scope of this guide.

Fetch the Raspberry Pi Firmware

The Raspberry Pi Firmware is maintained in a git repository. You will need to install git if you don't have it.

root #emerge -av git

Don't do anything you don't need to do as root.

As your normal user, make some space in the home directory for raspberry pi

user $mkdir raspberrypi
will do nicely.

This will be for the Raspberry Pi Firmware and kernel

user $cd raspberrypi

will fetch the Pi firmware into a directory called firmware

It will actually fetch all the history too. If you are short of space or bandwidth, a shallow clone is all that's needed.

Important
Receiving objects: 100% (484760/484760), 6.32 GiB

A full clone is 6.3G and growing, we only need the /boot directory out of the master branch.

There is nothing to build. ~/raspberrypi/firmware/boot is used as is.

Fetch, Configure and Build the Raspberry Pi Kernel

Fetch the Raspberry Pi Kernel

Stay in raspberrypi and

will fetch the kernel into a directory called linux.

With absolutely no fanfare at all, 64 bit support was added to this kernel tree late in 2016. No more searching for odd patches.

Not everything has been accepted by the mainline kernel yet but its getting closer. Feel free to test for yourself.

The master branch may be broken as its commit by commit as it happens. Test if you want to. Its better to use an identified tag. At the time of writing that is rpi-4.10.y. The 4.10 mainline kernel is still at -rc status.

user $cd linux
user $git checkout rpi-4.10.y

Checking out files: 100% (33079/33079), done.

Branch rpi-4.10.y set up to track remote branch rpi-4.10.y from origin.

Switched to a new branch 'rpi-4.10.y'

That's the kernel source tree in place, ready for configuring and cross compiling.

Configure The Kernel

Important
ARCH=arm64 must be specified everywhere or the kernel build system will use the HOST arch. This will destroy the arm64 .config file
As your user
user $cd raspberrypi/linux

The bcmrpi3_defconfig is almost right as it stands. It defaults to the powersave CPU governor, which runs the Pi at 600MHz. All the governors are there. Ondemand is recommended.

user $ARCH=arm64 CROSS_COMPILE=aarch64-unknown-linux-gnu- make bcmrpi3_defconfig

Either use menuconfig to change the default CPU governor, add something to the kernel command line, or change it after booting.

user $ARCH=arm64 CROSS_COMPILE=aarch64-unknown-linux-gnu- make menuconfig

Search for CPU_FREQ_DEFAULT_GOV go to that location and set the default to Ondemand. Exit menuconfig, saving the change.

.config - Linux/arm64 4.10.0-rc6 Kernel Configuration
 > CPU Power Management > CPU Frequency scaling ──────────────────────────────────
  ┌────────────────────────── CPU Frequency scaling ───────────────────────────┐
  │  Arrow keys navigate the menu.  <Enter> selects submenus ---> (or empty    │  
  │  submenus ----).  Highlighted letters are hotkeys.  Pressing <Y> includes, │  
  │  <N> excludes, <M> modularizes features.  Press <Esc><Esc> to exit, <?>    │  
  │  for Help, </> for Search.  Legend: [*] built-in  [ ] excluded  <M> module │  
  │ ┌────────────────────────────────────────────────────────────────────────┐ │  
  │ │    [*] CPU Frequency scaling                                           │ │  
  │ │    [*]   CPU frequency transition statistics                           │ │  
  │ │    [ ]     CPU frequency transition statistics details                 │ │  
  │ │          Default CPUFreq governor (powersave)  --->                    │ │  
  │ │    <*>   'performance' governor                                        │ │  
  │ │    -*-   'powersave' governor                                          │ │  
  │ │    <*>   'userspace' governor for userspace frequency scaling          │ │  
  │ │    <*>   'ondemand' cpufreq policy governor                            │ │  
 

The kernel .config contains lots of support for hardware you don't have and possibly have never heard of. More confident readers may be tempted to trim things out now. A word of advice - don't, at least, not until the system boots.

Cross Compiling The Kernel

The kernel will not link with the gold linker. It ends with

aarch64-unknown-linux-gnu-ld: fatal error: -shared and -pie are incompatible

If you don't know what the gold linker is, you are not using it.

The build is conventional, other than telling the build system to build for arm64 and use the cross compiler.

user $ARCH=arm64 CROSS_COMPILE=aarch64-unknown-linux-gnu- make -jX

Change X to the number of parallel MAKE jobs you want to run. Convention is cores+1.

Partition the microSD card

Warning
This Step Will Destroy ALL The Data on the Card

Background

The Pi does not need a boot loader. There is no firmware for the arm CPU, like there is in a PC. Instead, the GPU manages loading software for the CPU to execute, while the CPU is held reset. Its the GPU that has the 'BIOS' that gets everything started.

This makes it impossible to brick the Pi, since at worst, the microSD card needs to be reloaded.

This arrangement does impose some constraints on the microSD card.

The partition table must be MSDOS
The first partition (/boot) must be vfat.

The bootcode.bin file has some useful defaults that make setup easier, which we will take advantage of later.

Partitioning

Depending on how the microSD card is connected to you PC, it my be /dev/sdX or /dev/mmcblkY

Warning
Check! Do Get It Right - Don't Ruin your PC Install

In the example below, its /dev/sdk.

Using the partitioning tool of your choice, make three partitions on your microSD card.

boot 128Mb
swap 2G
root (/) the rest.

Using fdisk, and your microSD card block device, not my /dev/sdk

root #fdisk /dev/...

Make The Partition Table and Add Partitions

Make a new MSDOS disklable

Command (m for help): o

Add a new partition - this will be 128Mb for /boot

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-15523839, default 2048): 
Last sector, +sectors or +size{K,M,G,T,P} (2048-15523839, default 15523839): +128M

Has created a new partition 1 of type 'Linux' and of size 128 MiB.

Add a 2G partition 2 for swap.

Put the remaining space in partition 3 for root.

Check with

Command (m for help): p

Using an 8G microSD card it should show

Device     Boot   Start      End  Sectors  Size Id Type
/dev/sdk1          2048   264191   262144  128M 83 Linux
/dev/sdk2        264192  4458495  4194304    2G 83 Linux
/dev/sdk3       4458496 15523839 11065344  5.3G 83 Linux


Set Flags and Partition Types

Toggle the bootable flag on partition 1

Command (m for help): a
Partition number (1-3, default 3): 1

The bootable flag on partition 1 is enabled now.

Mark partition 1 as FAT

Command (m for help): t
Partition number (1-3, default 3): 1
Partition type (type L to list all types): c

Changed type of partition 'Linux' to 'W95 FAT32 (LBA)'.


Important
The Pi checks the type of partition 1 in the partition table and will not boot if its 82

Mark partition 2 as swap

Command (m for help): t
Partition number (1-3, default 3): 2
Partition type (type L to list all types): 82

Changed type of partition 'Linux swap / Solaris' to 'Linux swap / Solaris'.

Check again.

Device     Boot   Start      End  Sectors  Size Id Type
/dev/sdk1  *       2048   264191   262144  128M  c W95 FAT32 (LBA)
/dev/sdk2        264192  4458495  4194304    2G 82 Linux swap / Solaris
/dev/sdk3       4458496 15523839 11065344  5.3G 83 Linux


Save The New Partiton Table

Warning
Nothing has been written to the microSD card yet. This is the last opportunity to back out

Exit fdisk with either w or q

  w   write table to disk and exit
  q   quit without saving changes

Make Filesystems

You know the block devices from the partitioning step above. Fill in the ... to match your system.

vfat for /boot

root #mkfs -t vfat -F 32 /dev/...1
Important
Be sure to add the -F 32 switch. When using mkfs to make a vfat file system and -F 32 is not explicitly defined, mkdosfs will automatically select between 12, 16 and 32 bit, whatever mkdosfs thinks will fit better for the file system size, which in this case it would default to FAT16. This partition MUST be formatted FAT32 in order for the Raspberry Pi to boot.

mkswap for swap

root #mkswap /dev/...2

ext4 for root

root #mkfs -i 8192 -t ext4 /dev/...3
Important
-i 8192 makes an i-node for every two filesystem blocks, 8kB

The i-node count cannot be changed after filesystem creation and can limit the number of files on a files system. The Gentoo repository alone needs over 17,000 i-nodes.

Fetch the Gentoo bits of the install

To make it easy to cross refer to the Gentoo_Handbook

root #mkdir /mnt/gentoo

Mount the microSD card root filesystem at /mnt/gentoo

root #mount /dev/xxx3 /mnt/gentoo

Install the arm64 Stage 3

Following the Gentoo_Handbook fetch the arm64 stage3 and untar it to /mnt/gentoo in the normal way.

/mnt/gentoo/tmp should be empty. Clear it now or the the Pi will do it at boot. That may take a long time.

root #rm -rf /mnt/gentoo/tmp/*

Before and after clearing /mnt/gentoo/tmp

Filesystem                1K-blocks      Used Available Use% Mounted on
/dev/sdk3                   5380212   1205968   3881228  24% /mnt/gentoo
/dev/sdk3                   5380212    874636   4212560  18% /mnt/gentoo

Install a Portage Snapshot

This step is not actually needed to boot the Pi but emerge won't work without it

Following the Gentoo_Handbook fetch and unpack a portage shapshot in the normal way.

Careful readers can copy their host /usr/portage as long as ./packages and ./distfiles are omitted.

Populating /boot

The Raspberry Pi Foundation Provided Files

Mount the microSD card boot at /mnt/gentoo/boot. (The microSD card root should still be mounted at /mnt/gentoo)

root #mount /dev/xxx1 /mnt/gentoo/boot

As root, copy the content of your normal users ~/raspberrypi/firmware/boot to /mnt/gentoo/boot.

root #cp -a /home/<user>/raspberrypi/firmware/boot/* /mnt/gentoo/boot

This will generate a warning

cp: failed to preserve ownership for '<filename>': Operation not permitted

for every file. vfat does not support permissions, so the warning are expected and can be ignored.

You should end up with files in /mnt/gentoo/boot, not a directory called boot.

Not all the files there are required. Its a one size fits all for all models of Raspberry Pi operating in 32 bit mode.

Install the Kernel to the microSD Card

The kernel was built above, now to install it.

The kernel is in three parts

kernel binary
kernel modules
the device tree 

Install The Kernel Binary

As root, copy the kernel binary from the build location

root #cp /home/<user>/raspberrypi/linux/arch/arm64/boot/Image /mnt/gentoo/boot/kernel8.img
Important
If the file /boot/kernel8.img exists the bootcode.bin sets up the Pi for 64 bit operation and loads /boot/kernel8.img as the kernel

It is possible to use other kernel file names by adding entries to /boot/config.txt

Install The Device Tree

The device tree binary (.dtb) describes the hardware to the kernel. This avoids having all the existing hardware configurations ard coded into the kernel.

Due to the way the Raspberry Pi 64 bit kernel support has been added, there are going to be two different device trees with the same file name. A 32 bit version and a 64 bit version. They are not interchangeable. Move the 32 bit version out of the way.

root #mv /mnt/gentoo/boot/bcm2710-rpi-3-b.dtb /mnt/gentoo/boot/bcm2710-rpi-3-b.dtb_32
That's renamed the 32 bit version.

Copy the dtb from the build location

root #cp /home/<user>/raspberrypi/linux/arch/arm64/boot/dts/broadcom/bcm2710-rpi-3-b.dtb /mnt/gentoo/boot

It is possible to use other device tree file names by adding entries to /boot/config.txt

Install The Kernel Modules

From the top of the kernel tree, install the kernel modules.

root #cd /home/<user>/raspberrypi/linux
root #ARCH=arm64 CROSS_COMPILE=aarch64-unknown-linux-gnu- make modules_install INSTALL_MOD_PATH=/mnt/gentoo

The INSTALL_MOD_PATH is the root of the filesystem the modules are to be installed into. Due to kernel naming

user $uname -a

Linux Pi3 64bit 4.10.0-rc6-v8+ #2 SMP PREEMPT Thu Feb 2 20:34:34 GMT 2017 aarch64 GNU/Linux

its unlikely you will have a 4.10.0-rc6-v8+ so omitting INSTALL_MOD_PATH is probably harmless to your build host install.

Checking The Kernel Install

root #ls /mnt/gentoo/boot
COPYING.linux           bcm2708-rpi-cm.dtb      bcm2710-rpi-cm3.dtb  fixup_db.dat  kernel8.img   start_db.elf
LICENCE.broadcom        bcm2709-rpi-2-b.dtb     bootcode.bin         fixup_x.dat   overlays      start_x.elf
bcm2708-rpi-b-plus.dtb  bcm2710-rpi-3-b.dtb     fixup.dat            kernel.img    start.elf
bcm2708-rpi-b.dtb       bcm2710-rpi-3-b.dtb_32  fixup_cd.dat         kernel7.img   start_cd.elf

Notice the two bcm2710-rpi-3-b.dtb device tree binaries and the kernel8.img.

root #ls /mnt/gentoo/lib/modules
4.10.0-rc6-v8+

Shows that the kernel modules were installed to the correct location.

Fetching and Installing Firmware

Install WiFi Firmware

The on board wifi needs firmware. The files brcmfmac43430-sdio.txt and brcmfmac43430-sdio.bin are both required to be present in /mnt/gentoo/lib/firmware/brcm/

They can be downloaded via wireless.wiki.kernel.org

Install Bluetooth Firmware

The on board bluetooth needs the firmware files bcm43xx-0.fw and bcm43xx_hdr-0.fw in /mnt/gentoo/lib/firmware.

It also needs a startup script in /mnt/gentoo/etc/local.d/bluetooth.start

#!/bin/bash
# Bluetooth is attached to /dev/ttyAMA0 so attach it and 
# load the firmware 
/usr/bin/hciattach /dev/ttyAMA0 bcm43xx 921600 noflow -

Setup

Much of the setup prior to booting a new Gentoo install is done in a chroot. Chrooting into an arm64 install from some other arch is beyond the scope of this guide. Instead, only the bare minimum setup, from outside the chroot is covered.

Root Password

Important
You need this to be able to log in at all

There are several ways to generate a password hash for /etc/shadow I usually copy the hash from another system. My Pi uses an /etc/shadow root entry

root:$6$xxPVR/Td5iP$/7Asdgq0ux2sgNkklnndcG4g3493kUYfrrdenBXjxBxEsoLneJpDAwOyX/kkpFB4pU5dlhHEyN0SK4eh/WpmO0::0:99999:7:::

The matching password is raspberry.

Feel free to use that line to replace the root entry in /mnt/gentoo/etc/shadow

You can change the password once you are logged in.

/etc/fstab

On the Pi, the microSD card will be /dev/mmcblk0 with partitions /dev/mmcblk0p1, /dev/mmcblk0p2 and /dev/mmcblk0p3

Edit /mnt/gentoo/etc/fstab to match.

root #nano -w /mnt/gentoo/etc/fstab
/dev/mmcblk0p1          /boot           vfat            noauto,noatime  1 2
/dev/mmcblk0p2          none            swap            sw              0 0
/dev/mmcblk0p3          /               ext4            noatime         0 1

/boot/config.txt

This file will not exist until its created. Create it with the following content.

root #nano -w /mnt/gentoo/boot/config.txt
# have a properly sized image

disable_overscan=1

# lets have the VC4 hardware accelerated video

dtoverlay=vc4-fkms-v3d

# for sound over HDMI

hdmi_drive=2

# Enable audio (loads snd_bcm2835)

dtparam=audio=on

# gpu_mem is for closed-source driver only; since we are only using the
# open-source driver here, set low

gpu_mem=16

/boot/cmdline.txt

This file will not exist until its created. Create it with the following content.

root #nano -w /mnt/gentoo/boot/cmdline.txt

root=/dev/mmcblk0p3 rootfstype=ext4 rootwait

This boots the Pi with a kernel command line of

8250.nr_uarts=0 cma=256M@256M dma.dmachans=0x7f35 bcm2708_fb.fbwidth=1920 bcm2708_fb.fbheight=1080 bcm2709.boardrev=0xa02082 bcm2709.serial=0x8e2830fe smsc95xx.macaddr=B8:27:EB:28:30:FE bcm2708_fb.fbswap=1 bcm2709.uart_clock=48000000 vc_mem.mem_base=0x3dc00000 vc_mem.mem_size=0x3f000000  root=/dev/mmcblk0p3 rootfstype=ext4 rootwait

It really is one long line.

Setting the Console Keymap

This step is optional if you can log in using the default

keymap="us"
root #nano -w /mnt/gentoo/etc/conf.d/keymaps

Set the keymap to something you use, e.g.

keymap="dvorak-uk"

Boot the Pi to Test

Unmount the microSD card.

root #umount /mnt/gentoo/boot
root #umount /mnt/gentoo/

When the prompt returns, move the microSD card to the Raspberry Pi and power on.

For 10 seconds (it seems much longer) you should see the GPU 'Rainbow' test pattern, then the familiar boot messages.

Log in at the Pi console. Nothing was added to any runlevels during the install, so networking was not started, nor anything that depends on networking, like ntpd and sshd.

The Pi does not have a hardware real time clock. Its time will be Jan 1, 1970.

What Next

As always with Gentoo, if it booted, that's the hard bit done.

All The setup steps in the Gentoo Handbook
Fix the MAC address or use a static IP
Allow root logins via ssh 
Add a crond, a logger and other things the handbook does before the reboot. 
Add Kernel Sources (or at least the .config)

Random Hints

WiFi and Bluetooth

Its unlikely that WiFi or Bluetooth will work at first boot. Expect to add some control tools for both.


CFLAGS

CFLAGS="-march=armv8-a+crc -mtune=cortex-a53 -ftree-vectorize -O2 -pipe -fomit-frame-pointer"

gcc-6.x allows the use of -march=native but that will prevent the use of distcc. The above is the same as gcc-6.3 would set for -march=native anyway.

ACCEPT_KEYWORDS

Outside of the @system set, arm64 is either testing or keyword masked. Set

ACCEPT_KEYWORDS="~arm64"

and expect to use package.accept_keywords too.

Warning
The stage 3 tarball has been built with gcc-4.9 as gcc-5.4 is in testing ...

Upgrade gcc then rebuild all of the installed C++ software.

Important
News item 2015-10-22 GCC 5 Defaults to the New C++11 ABI

MAKEOPTS

With only 1G RAM, and four cores, the conventional

MAKEOPTS="-j5"

is a bit aggressive for building larger things. It will force swapping or even appear to lock up the Pi completely, to the point where it won't even respond to the console.

Use files in /etc/portage/env/ and entries in /etc/portage/package.env to set MAKEOPTS on a per package basis.

Networking

dhcpcd is not in the stage 3, nor is eth0 in the default runlevel.

root #busybox udhcpc eth0
will bring up eth0

sshd

The default configuration for sshd will not allow password based root logins.

add your ssh public key for root
make a normal user in the wheel group
edit /etc/ssh/sshd_config to allow password based root logins

Where to Get Help

On Internet Relay Chat

irc.freenode.net#gentoo-arm
irc.freenode.net#gentoo-embedded

On the Gentoo Forums, start a new topic in the Gentoo on Alternative Architectures forum.

I don't mind a PM on the forums with a link to your post. I don't do one to one help via email or the forums PM system. You will either get no response at all or a request to make a public post. That way others may learn from your misfortune.

Acknowledgements

Everyone contributing to the arm64 software base.

Especially Sakaki, who showed the way on the final steps.