User:Rebeltrouper/Raspberry Pi4 and PI400 64 bit install

From Gentoo Wiki
Jump to:navigation Jump to:search

The Raspberry Pi 4 increase in power has made it a popular choice for specific tasks. These can include hosting servers, samba shares, streaming devices, and many more. The main problem is the weak hardware of the Raspberry Pi when compared to amd64 CPUs. The main point of this guide is to install Gentoo on a Raspberry Pi using crossdev and qemu to speed up the compiling process for bootstrapping. Then set up a binhost your Pi so that it can download from you amd64 computer. The Pi itself is too slow to do compiling, unless you want to wait days for something to finish. Finally setting up X in your Pi to show off.

This process will be bootstrapping a Gentoo install from a stage2 that you create. Some additional steps will be outlined in making it a stage3 then a stage4. This will be more for an advanced user as certain things can get weird. If you want the easiest way to install and slow way to install Gentoo will be listed later on.

Setup using a normal way that gets complicated. later on a different way where crossdev is built inside a chroot

Crossdev has a problem where it will confuse your make.conf with the target causing failure. The best way to do this is to set up a chroot environment just for the crossdev. After you set it up and compile crossdev and things like that, you will actually delete your make.conf in the chroot and the make.profile.

The best way I can think of is doing this in a VM and go to part2 first and then do the steps of part 1. However a little more knowledge will be needed in how a system works though. But the speed up doing that way can be over 10 fold.

Install crossdev on the PC

An initial set up for the pi is needed in order for it to boot. The PI will need a toolchain in order to build the rest of the system. Crossdev creates a toolchain on your main amd64 computer to cross-compile the code into aarch64/arm64 executable code. This will give a major speed boost over trying to do this on the pi itself and should be the main attempt in install and maintain the pi. To understand the complexity of crossdev, one should read the section on crossdev itself as some familiarity will be assumed from this point on.

1emerge --ask sys-devel/crossdev
2crossdev --stable -t aarch64-unknown-linux-gnu
Important
Be warned that crossdev needs to be set up before these steps correctly. There is the possibility of it polluting your system and corrupting it. Please look at the crossdev guide on gentoo to set it up correctly

It will compile the tools needed for your system to build arm64 inside of a user group /usr/aarch64-unknown-linux-gnu. From now on in the crossdev system this is where you will be building things in your system.

kernel compiling and modules

Lastly for the initial set up you will needed git and to fetch the Raspberry Pi kernel to compile. You will first create a directory for the pi kernel to exist and will compile it. This will create a temp directory and put everything in there. Then it will git the Raspberry Pi kernel sources, compile the correct config setupt for them, allow modifications with menuconfig, compile an image and modules and dtbs for the pi to work, and finally move them into a boot folder to just copy over to your SD card.

emerge --ask dev-vcs/git

mkdir --parents ~/opt/raspberry
mkdir ~/opt/raspberry-pi-boot
mkdir ~/opt/raspberry-pi-boot/overlays
cd ~/opt/raspberry
git clone -b stable --depth=1 https://github.com/raspberrypi/linux
cd linux
KERNEL=kernel8
make -j <nproc> ARCH=arm64 CROSS_COMPILE=aarch64-unknown-linux-gnu- bcm2711_defconfig
make -j <nproc> ARCH=arm64 CROSS_COMPILE=aarch64-unknown-linux-gnu- menuconfig
make -j <nproc> ARCH=arm64 CROSS_COMPILE=aarch64-unknown-linux-gnu- Image modules dtbs
make -j <nproc> ARCH=arm64 CROSS_COMPILE=aarch64-unknown-linux-gnu- INSTALL_MOD_PATH=../modules modules_install
cp ~/opt/raspberry/arch/arm64/boot/Image ~/opt/raspberry-pi-boot/Kernel8.img
cp ~/opt/raspberry/arch/arm64/boot/dts/broadcom/*.dtb ~/opt/raspberry-pi-boot/
cp ~/opt/raspberry/arch/arm64/boot/dts/overlays/*.dtb ~/opt/raspberry-pi-boot/overlays

Now the kernel is done and will focus on the hard part of getting your system stage2 working. At the end booting will be discussed as arm does not boot the same as amd64.

Stage2 creation

You now should have a directory /usr/aarch64-unknown-linux-gnu. The crossdev has build a tool-chain for it to use but not the tool-chain that the system needs needs. From this point on you can use emerge as you would normally but instead replace it with emerge-aarch64-unknown-linux-gnu and will use crossdev to compile it into /usr/aarch64-unknown-linux-gnu. Now we need to build the pi actual system.

First edit your make.conf file in /usr/aarch64-unknown-linux-gnu/etc/portage/make.conf and add any optimizations you would like. One recommended one is -mcpu=cortex-a72.

An example of my make.conf is below. The two CFLAGS is that some packages will not compile with corssdev and the first debug should be to remove -mcpu=cortex-a72.

# Note: profile variables are set/overridden in profile/ files:
# etc/portage/profile/use.force (overrides kernel_* USE variables)
# etc/portage/profile/make.defaults (overrides ARCH, KERNEL, ELIBC variables)

CHOST=aarch64-unknown-linux-gnu
CBUILD=x86_64-pc-linux-gnu

ROOT=/usr/${CHOST}/

ACCEPT_KEYWORDS="${ARCH}" #~${ARCH}"

USE="${ARCH} X pulseaudio -acl -cups"

#CFLAGS="-O2 -pipe -fomit-frame-pointer"
CFLAGS="-O2 -pipe -fomit-frame-pointer -mcpu=cortex-a72 "

CXXFLAGS="${CFLAGS}"

MAKEOPTS="-j10"
ACCEPT_KEYWORDS="arm64"
EMERGE_DEFAULT_OPTS="--ask --verbose"
VIDEO_CARDS="v3d vc4"
LICENSE_ACCEPT="*"

FEATURES="-collision-protect sandbox buildpkg noman noinfo nodoc"
BINPKG_COMPRESS="lz4"

# Be sure we dont overwrite pkgs from another repo..
PKGDIR=${ROOT}var/cache/binpkgs/
PORTAGE_TMPDIR=${ROOT}tmp/

PKG_CONFIG_PATH="${ROOT}usr/lib/pkgconfig/"
#PORTDIR_OVERLAY="/var/db/repos/local/"

Now you need to build the tool-chain.

#select a make.profile with either ln -s or eselect
ARCH=arm64 PORTAGE_CONFIGURE_ROOT=/usr/aarch64-unknown-linux-gnu eselect profile list

#rebuild a fresh system. Most packages will be missing
emerge-aarch64-unknown-linux-gnu --keep-going @system

#emerge-aarch64-unknown-linux-gnu --sync && emerge-aarch64-unknown-linux-gnu -uDN --keep-going @world
#Depending on what you decided to do with your system and make.conf in your host, some packages may fail.
Warning
If you move your host OS make.conf to be make.conf.bak.bak you will get no errors during compiling and crossdev will work. If you do not wish do this you will need to chroot into the install to finish making it. Some packages will fail.

Since you are missing most folders in your system, you will have to make them. Some things you will need to remember to add are below.

  • package.use
  • package.accept
  • /home
  • fstab (although it should be done later)
  • proc
  • dev
  • sys
Warning
This should build without error. However there is a problem with crossdev at this point and some packages will fail to compile. This is fine for now as you will chroot in to fix all the rest.

Binfmt setup

In order to chroot into arm64, you will need the instruction set. Pleases look at binfmt for openrc and systemd to get it working on your system.

For systemd you can use the following commands

Chroot into your system to finish building

Crossdev has a problem as stated earlier and thus the only sane way to build anything at this point is to chroot into the system. This is much slower than using crossdev but the only safe way to know things will work. Most X packages will not compile however crossdev should work fine if you are making this headless.

First the system must be prepared to be chroot into it and will have some bugs that need fixed. First need to put qemu inside of the chroot.

cd /usr/aarch64-unknown-linux-gnu
ROOT=$PWD/ emerge -K qemu-user

Now to change your make.conf file for chroot.

1cd /usr/aarch64-unknown-linux-gnu/etc/portage
2mv make.conf make.conf.crossdev
3vim make.conf

Now add this to your new make.conf

# Note: profile variables are set/overridden in profile/ files:
# etc/portage/profile/use.force (overrides kernel_* USE variables)
# etc/portage/profile/make.defaults (overrides ARCH, KERNEL, ELIBC variables)



#CFLAGS="-O2 -pipe -fomit-frame-pointer"
CFLAGS="-O2 -pipe -fomit-frame-pointer -mcpu=cortex-a72 "

CXXFLAGS="${CFLAGS}"

MAKEOPTS="-j10"
ACCEPT_KEYWORDS="aarch64"
ACCEPT_LICENSE="*"

PORTAGE_BINHOST="ssh://localhost@10.0.0.121/usr/aarch64-unknown-linux-gnu/var/cache/binpkgs/"


FEATURES="-collision-protect -pid-sandbox -network-sandbox buildpkg noman noinfo nodoc getbinpkg"
BINPKG_COMPRESS="lz4"
# Be sure we dont overwrite pkgs from another repo..
PKGDIR=${ROOT}var/cache/binpkgs/
PORTAGE_TMPDIR=${ROOT}tmp/

#PKG_CONFIG_PATH="${ROOT}usr/lib/pkgconfig/"
#PORTDIR_OVERLAY="/var/db/repos/local/"
EMERGE_DEFAULT_OPTS="$EMERGE_DEFAULT_OPTS --exclude app-emulation/qemu"
Warning
QEMU chroot will not work unless you turn off -pid-sandbox. If you are not able to chroot the mostly likely problem is that the qemu you put in there does not have user/soft targets with aarch64.

There is one more thing that need to be done to emerge itself and to set up your profile.

cd /usr/aarch64-unknown-linux-gnu/etc/portage
ln -s make.profile ../../var/db/repos/gentoo/profiles/default/linux/arm64/17.0/<desired one>

vim /usr/aarch64-unknown-linux-gnu/usr/bin/emerge
change the top line #!/usr/bin/python-exec to #!/usr/bin/python3.10
#you can have a different version of python
Warning
From this point on you must fix portage. It is described above. It will not compile anymore and some updates will revert it. So if you get weird errors about python-exec not working, you will need to go back and change it again. This will happen on portage updates.

This is because an error will occur and to link the python correctly to portage.

#do what is needed to chroot into your aarch64 system 
#mount ....
chroot /usr/aarch64-unknown-linux-gnu/. /bin/bash --login


emerge --sync

emerge -uDN @world
#this will take a while.
#here are some additional packages you may want as well.
emerge -uDN @world dosfstools dhcpcd raspberrypi-firmware

make a stage4 at this point with tar or a script you can find online.

It now is close to a stage3 and can follow something like the amd64 install for steps similar to after the chroot.

Chroot advice and speed ups

The chroot system is by far the best way to get things to compile when you are having problems. It is very slow though. One good strategy to get the system to work is to binhost from your own system. The chroot will have nothing in its world and it will want to download and redo every package. It will try and redo things even if you cross-compiled it. So setting up a binhost to download from itself is by far the best way to speed this up. You can cross-compile all the packages you are able to and refinish compiling it in the chroot and not have to redo any steps.

Understanding chroot and crossdev problems

Certain packages will just not compile unless you are very good at coding and can fix them yourself. I am not that good and my solution is to use package.mask liberally some times. One example is that python3.11 will not compile so I masked it and force all programs to use python3.10 instead. Also make sure to check that you are install stable branches and not testing branches. For some reason th chroot likes to install testing versions.

create your SD card and move the items over

Booting from a pi is different than amd64. I believe the pi4 is mostly hardcoded to boot form kernel8.img and it reads a config.txt and cmdline.txt for how to do the boot. An example I use is from Raspbian. If you wish to boot from another kernel, you will have to set kernel= in your config.txt

config.txt

# For more options and information see
# http://rpf.io/configtxt
# Some settings may impact device functionality. See link above for details

# uncomment if you get no picture on HDMI for a default "safe" mode
#hdmi_safe=1

# uncomment the following to adjust overscan. Use positive numbers if console
# goes off screen, and negative if there is too much border
#overscan_left=16
#overscan_right=16
#overscan_top=16
#overscan_bottom=16

# uncomment to force a console size. By default it will be display's size minus
# overscan.
#framebuffer_width=1280
#framebuffer_height=720

# uncomment if hdmi display is not detected and composite is being output
#hdmi_force_hotplug=1

# uncomment to force a specific HDMI mode (this will force VGA)
#hdmi_group=1
#hdmi_mode=1

# uncomment to force a HDMI mode rather than DVI. This can make audio work in
# DMT (computer monitor) modes
#hdmi_drive=2

# uncomment to increase signal to HDMI, if you have interference, blanking, or
# no display
#config_hdmi_boost=4

# uncomment for composite PAL
#sdtv_mode=2

#uncomment to overclock the arm. 700 MHz is the default.
#arm_freq=800

# Uncomment some or all of these to enable the optional hardware interfaces
#dtparam=i2c_arm=on
#dtparam=i2s=on
#dtparam=spi=on

# Uncomment this to enable infrared communication.
#dtoverlay=gpio-ir,gpio_pin=17
#dtoverlay=gpio-ir-tx,gpio_pin=18

# Additional overlays and parameters are documented /boot/overlays/README

# Enable audio (loads snd_bcm2835)
dtparam=audio=on

# Automatically load overlays for detected cameras
camera_auto_detect=1

# Automatically load overlays for detected DSI displays
display_auto_detect=1

# Enable DRM VC4 V3D driver
dtoverlay=vc4-kms-v3d
max_framebuffers=2

# Run in 64-bit mode
arm_64bit=1

# Disable compensation for displays with overscan
disable_overscan=1

[cm4]
# Enable host mode on the 2711 built-in XHCI USB controller.
# This line should be removed if the legacy DWC2 controller is required
# (e.g. for USB device mode) or if USB support is not required.
otg_mode=1

[all]

[pi4]
# Run as fast as firmware / board allows
arm_boost=1

[all]

cmdline.txt

dwc_otg.lpm_enable=0 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 cgroup_enable=memory elevator=deadline rootwait

Create your USB device you want and read what other people have done. There are optimizations and things like that can be important.

#exit your chroot at this point after the stage4. You will be generating your pi at this point.

sudo mkdir /mnt/pi-mount
sudo mount /dev/sdb2 /mnt/pi-mount
sudo mv /usr/aarch64-unknown-linux-gnu/stage4.tar.gz /mnt/pi-mount/.
cd /mnt/pi-mount
sudo tar -xvf stage4.tar.gz --numeric-owner --xattrs-include='*.*'
sudo mount /dev/sda1 boot
sudo cp -r ~/opt/raspberry-pi-boot/* /mnt/pi-mount/boot/.
sudo cp -r ~/opt/modules/<Kernel-version> /mnt/pi-mount/usr/lib/modules/.

Now your boot and your modules and your root folder should be set up. Add the cmdline.txt and the config.txt to your mount boot.

This system should be bootable in its current state. Although it will only contain what you put in it for the stage4. Now to set up a binhost and change the pi make.conf.

Setting up binhost in your pi

At this point your pi will boot and you can proceed with everything inside the pi via ssh. The PI itself will need a new make.conf. The main point of this version is to finally remove all the qemu/crossdev from it and have it use packages.

My pi make.conf is show below.

# Note: profile variables are set/overridden in profile/ files:
# etc/portage/profile/use.force (overrides kernel_* USE variables)
# etc/portage/profile/make.defaults (overrides ARCH, KERNEL, ELIBC variables)



CFLAGS="-O2 -pipe -fomit-frame-pointer -mcpu=cortex-a72 "

CXXFLAGS="${CFLAGS}"

MAKEOPTS="-j4"
ACCEPT_KEYWORDS="arm64"
ACCEPT_LICENSE="*"
EMERGE_DEFAULT_OPTS="--ask --verbose"
FEATURES="getbinpkg"
L10N="en en-UTF8"
PORTAGE_BINHOST="ssh://<your-username>@<Your-local-ip>/usr/aarch64-unknown-linux-gnu/var/cache/binpkgs/"
VIDEO_CARDS="v3d vc4 egl gles1 gles2"
INPUT_DEVICES="libinput evdev"




USE="X pulseaudio xorg  -acl -cups"

#EMERGE_DEFAULT_OPTS="$EMERGE_DEFAULT_OPTS --exclude app-emulation/qemu"

The main thing for this make.conf is to set up where the PORTAGE_BINHOST will look. The easiest for me was to just have it scp it over and use ssh-agent and ssh-add for pass wordless authentication. You can run all the heavy compiling in the QEMU on your AMD64 and just pull the packages you want via this way.

In general it should just work and no python workarounds are needed. On the PI it is now a normal gentoo operating system.

The fastest and best way to compile but gets weird.

Crossdev without chroot is the best solution I can think of.

Crossdev has problems where it gets confused with your make.conf file of your real system and the crossdev make.conf. There is one solution and that is to build a VM or chroot env where you have only qemu/binfmt/crossdev inside of it. After the system is built and configured you can delete or rename your make.conf in your chroot/VM environment. This will let you use crossdev only in your system giving a MASSIVE speed up. Most packages will compile now unlike before but some will have problems. This gets weird in the sense that you may have a chroot inside of a chroot to do this.

However setting this up in some VM amd64 system following the system and deleting as a said would work the best and probably the most sane. Then you can set up a bridged network device in virt-manager and expose the VM to your network and binhost will work. So far this method has done me wonders as I have setup a binhost system via this method. Seems to have problems with vim/firefox though with crossdev.

Bridge eth0 with br0 to pass through with to virt-manager to expose VM to local network

Creating a VM seems to be a very good setup as a whole for doing this. You can "fix" a lot of the problems and still expose the device to the network.

To create your bridge do the following command.

sudo ip link add name br0 type bridge
sudo ip link set dev br0 up
sudo ip link set dev eth0 master br0

Now pass this over in virt-manager and do all steps in the first part. Your VM will be the entire section you want to deal with and rename your make.conf to make.conf.bak and cross-dev should work much better.

Installing inside the VM

Since the install is to be as minimal as possible, only get what is needed to boot, dhcpcd, crossdev, and qemu. After you build the system, you need to remove the make.conf file from your host system inside the VM.

mv /etc/portage/make.conf /etc/portage/make.conf.bak.bak

Currently testing

If the above method works I may post an iso file with it preconfigured so that it can just work. I guess docker would work better but I do not understand that as well. It should just be a run it in virt-manager and you can edit the size and other stuff.

Initial testing with the VM/crossdev is working very well. So far no problems and when you bridge you can have a real IP. This makes it the most advantageous way to do his. You can get a real system that you do not care if you update as much or not and have it be a host for building all your crossdev packages.

Next step of testing is thinking of a good way to sync /etc/portage with each other. If you change one it will auto update the other. Current thought process is to put a script in your VM and do all the compiling there. Then rsync that setup over to your pi.

Weird quarks that have arised.

World files will be different between chroot and emerge-aarch64

The world files will not be shared. So if you need to chroot in the system will have to reinstall everything as it gets confused. The packages should already be in there though, they just need to be understood that they exist. Maybe even binhosting to itself since all the packages exist. I think maybe setting up a build package may be good to fix this. Maybe ccache (I have never used this though).