User:Rebeltrouper/Raspberry Pi4 and PI400 64 bit install

From Gentoo Wiki
Jump to:navigation Jump to:search

The Raspberry Pi 4 increase in power has made it a popular choice for specific tasks. These can include hosting servers, samba shares, streaming devices, and many more. The main problem is the weak hardware of the Raspberry Pi when compared to amd64 CPUs. The main point of this guide is to install Gentoo on a Raspberry Pi using crossdev and qemu to speed up the compiling process for bootstrapping. Then set up a binhost so the PI can download from an amd64 computer. The Pi itself is too slow to do compiling for most programs as general tasks can take hours. Finally getting a working X env.

This process will be bootstrapping a Gentoo install from a stage2. Some additional steps will be outlined in making it a stage3 then a stage4. This will be more for an advanced user as certain things can get weird. If you want the easiest way to install and slow way to install Gentoo will be listed later on.

Setup using a normal way that gets complicated. later on a different way where crossdev is built inside a chroot

Crossdev has a problem were a specific package will put certain flags and compilier options from your host machine. This cannot really be fixed as it is auto probing your amd64 host machine. The best way to get by this is to compile as many features as you can on the amd64 machine and compile what fails on the aarch64 machine itself.

Install crossdev on the PC

An initial set up for the pi is needed in order for it to boot. The PI will need a toolchain in order to build the rest of the system. Crossdev creates a toolchain on the main amd64 computer to cross-compile the code into aarch64/arm64 executable code. This will give a major speed boost over trying to do this on the pi itself and should be the main attempt in install and maintaining the pi. To understand the complexity of crossdev, one should read the section on crossdev itself as some familiarity will be assumed from this point on. 

1emerge --ask sys-devel/crossdev
2crossdev --stable -t aarch64-unknown-linux-gnu
Important
Be warned that crossdev needs to be set up before these steps correctly. There is the possibility of it polluting your system and corrupting it. Please look at the crossdev guide on gentoo to set it up correctly

It will compile the tools needed and a base system and put them in the folder /usr/aarch64-unknown-linux-gnu. From now on in the crossdev system this is where all of the crossdev building will be put to and compiled into.

kernel compiling and modules

Lastly for the initial set up git will needed to fetch the Raspberry Pi kernel to compile. First create a directory for the pi kernel to exist and will compile it. This will create a temp directory and put everything in there. Then it will git the Raspberry Pi kernel sources, compile the correct config setup for them, allow modifications with menuconfig, compile an image and modules and dtbs for the pi to work, and finally move them into a boot folder to just copy over to your SD card. 

emerge --ask dev-vcs/git

mkdir --parents ~/opt/raspberry
mkdir ~/opt/raspberry-pi-boot
mkdir ~/opt/raspberry-pi-boot/overlays
cd ~/opt/raspberry
git clone -b stable --depth=1 https://github.com/raspberrypi/linux
cd linux
KERNEL=kernel8
make -j <nproc> ARCH=arm64 CROSS_COMPILE=aarch64-unknown-linux-gnu- bcm2711_defconfig
make -j <nproc> ARCH=arm64 CROSS_COMPILE=aarch64-unknown-linux-gnu- menuconfig
make -j <nproc> ARCH=arm64 CROSS_COMPILE=aarch64-unknown-linux-gnu- Image modules dtbs
make -j <nproc> ARCH=arm64 CROSS_COMPILE=aarch64-unknown-linux-gnu- INSTALL_MOD_PATH=../modules modules_install
cp ~/opt/raspberry/arch/arm64/boot/Image ~/opt/raspberry-pi-boot/Kernel8.img
cp ~/opt/raspberry/arch/arm64/boot/dts/broadcom/*.dtb ~/opt/raspberry-pi-boot/
cp ~/opt/raspberry/arch/arm64/boot/dts/overlays/*.dtb ~/opt/raspberry-pi-boot/overlays

Now the kernel is done and will focus on the hard part of getting your system stage2 working. At the end booting will be discussed as arm does not boot the same as amd64.

RPI4 upgrading kernel

The PI iteself can compile its own kernel after it is bootstrapped. It differs slightly than the above method. Below is an example of upgrading a kernel by using the RP4s 4 cores.

cd /usr/src/linux

#use config file in /proc
zcat /proc/config.gz .config
make oldconfig

#or use default config
make defconfig

#Now to build kernel and modules
make menuconfig
make -j4 Image dtbs modules #takes about 6 hours.
make dtbs_install
make modules_install
make install
cp /boot/kernel8.img /boot/kernel8.img.bak
cp arch/arm64/boot/Image /boot/kernel8.img

Stage2 creation

Now the directory /usr/aarch64-unknown-linux-gnu should exist. The crossdev has build a tool-chain for it to use but not the tool-chain that the system needs. From this point on emerge can be used as normal emerge but instead replace it with emerge-aarch64-unknown-linux-gnu and will use crossdev to compile it into /usr/aarch64-unknown-linux-gnu. Now we need to build the pi actual system.

First edit your make.conf file in /usr/aarch64-unknown-linux-gnu/etc/portage/make.conf and add any optimizations you would like. One recommended one is -mcpu=cortex-a72.

An example of my make.conf is below. The two CFLAGS is that some packages will not compile with corssdev and the first debug should be to remove -mcpu=cortex-a72.

# Note: profile variables are set/overridden in profile/ files:
# etc/portage/profile/use.force (overrides kernel_* USE variables)
# etc/portage/profile/make.defaults (overrides ARCH, KERNEL, ELIBC variables)

CHOST=aarch64-unknown-linux-gnu
CBUILD=x86_64-pc-linux-gnu

ROOT=/usr/${CHOST}/

ACCEPT_KEYWORDS="${ARCH}" #~${ARCH}"

USE="${ARCH} X pulseaudio -acl -cups"

#CFLAGS="-O2 -pipe -fomit-frame-pointer"
CFLAGS="-O2 -pipe -fomit-frame-pointer -mcpu=cortex-a72 "

CXXFLAGS="${CFLAGS}"

MAKEOPTS="-j10"
ACCEPT_KEYWORDS="arm64"
EMERGE_DEFAULT_OPTS="--ask --verbose"
VIDEO_CARDS="v3d vc4"
LICENSE_ACCEPT="*"

FEATURES="-collision-protect sandbox buildpkg noman noinfo nodoc"
BINPKG_COMPRESS="lz4"

# Be sure we dont overwrite pkgs from another repo..
PKGDIR=${ROOT}var/cache/binpkgs/
PORTAGE_TMPDIR=${ROOT}tmp/

PKG_CONFIG_PATH="${ROOT}usr/lib/pkgconfig/"
#PORTDIR_OVERLAY="/var/db/repos/local/"

Now time to build the system and its toolchains. 

#select a make.profile with either ln -s or eselect
ARCH=arm64 PORTAGE_CONFIGURE_ROOT=/usr/aarch64-unknown-linux-gnu eselect profile list

#rebuild a fresh system. Most packages will be missing
emerge-aarch64-unknown-linux-gnu --keep-going @system

emerge-aarch64-unknown-linux-gnu -uDN --keep-going @world
#Depending on what you decided to do with your system and make.conf in your host, some packages may fail.
Warning
If you move your host OS make.conf to be make.conf.bak.bak you will get no errors during compiling and crossdev will work. If you do not wish do this you will need to chroot into the install to finish making it. Some packages will fail.

Below are some directories and files that will be missing since this is a stage2 creation.

  • package.use
  • package.accept
  • /home
  • fstab (although it should be done later)
  • proc
  • dev
  • sys
Warning
This should build without error. However there is a problem with crossdev at this point and some packages will fail to compile. This is fine for now as you will chroot in to fix all the rest.

Binfmt setup

In order to chroot into arm64, you will need the instruction set. Pleases look at binfmt on the gentoo wiki in order to complete the build and to chroot into the system. This is needed becasue the system is no longer amd64 but aarch64 and need the correct instruction set.

Chroot into your system to finish building

Crossdev has a problem as stated earlier and thus the only sane way to build anything at this point is to chroot into the system. This is much slower than using crossdev but the only safe way to know things will work. Most X packages will not compile however crossdev should work fine if you are making this headless.

First the system must be prepared to be chroot into it and will have some bugs that need fixed. First need to put qemu inside of the chroot. 

cd /usr/aarch64-unknown-linux-gnu
ROOT=$PWD/ emerge -K qemu-user

Now to change your make.conf file for chroot.

1cd /usr/aarch64-unknown-linux-gnu/etc/portage
2mv make.conf make.conf.crossdev
3vim make.conf

Below is a chroot make.conf example that has been used before and works. 

# Note: profile variables are set/overridden in profile/ files:
# etc/portage/profile/use.force (overrides kernel_* USE variables)
# etc/portage/profile/make.defaults (overrides ARCH, KERNEL, ELIBC variables)



#CFLAGS="-O2 -pipe -fomit-frame-pointer"
CFLAGS="-O2 -pipe -fomit-frame-pointer -mcpu=cortex-a72 "

CXXFLAGS="${CFLAGS}"

MAKEOPTS="-j10"
ACCEPT_KEYWORDS="aarch64"
ACCEPT_LICENSE="*"



FEATURES="-collision-protect -pid-sandbox -network-sandbox buildpkg noman noinfo nodoc getbinpkg"
BINPKG_COMPRESS="lz4"
# Be sure we dont overwrite pkgs from another repo..
PKGDIR=${ROOT}var/cache/binpkgs/
PORTAGE_TMPDIR=${ROOT}tmp/

#PKG_CONFIG_PATH="${ROOT}usr/lib/pkgconfig/"
#PORTDIR_OVERLAY="/var/db/repos/local/"
EMERGE_DEFAULT_OPTS="$EMERGE_DEFAULT_OPTS --exclude app-emulation/qemu"
Warning
QEMU chroot will not work unless you turn off -pid-sandbox. If you are not able to chroot the mostly likely problem is that the qemu you put in there does not have user/soft targets with aarch64.

There is one more thing that need to be done to emerge itself and to set up your profile. The profile in portage is not set and must be set manually. This can be done by linking the target profile of your choice and the make.profile 

cd /usr/aarch64-unknown-linux-gnu/etc/portage
ln -s make.profile ../../var/db/repos/gentoo/profiles/default/linux/arm64/17.0/<desired one>

vim /usr/aarch64-unknown-linux-gnu/usr/bin/emerge
change the top line #!/usr/bin/python-exec to #!/usr/bin/python3.10
#you can have a different version of python
Warning
From this point on you must fix portage. It is described above. It will not compile anymore and some updates will revert it. So if you get weird errors about python-exec not working, you will need to go back and change it again. This will happen on portage updates.

This is because an error will occur and to link the python correctly to portage. 

#do what is needed to chroot into your aarch64 system 
#mount ....
chroot /usr/aarch64-unknown-linux-gnu/. /bin/bash --login


emerge --sync

emerge -uUD @world
#this will take a while.
#here are some additional packages you may want as well.
emerge -uDN @world dosfstools dhcpcd raspberrypi-firmware

make a stage4 at this point with tar or a script you can find online.

It now is close to a stage3 and can follow something like the amd64 install for steps similar to after the chroot.

Chroot advice and speed ups

The chroot system is by far the best way to get things to compile when programs are having problems. It is very slow though. One good strategy to get the system to work is to binhost from your own system. So setting up a binhost to download from itself is by far the best way to speed this up. Cross-compile all the packages that can be compile and finish the rest in the chroot.

Understanding chroot and crossdev problems

Certain packages will just not compile unless you are very good at coding and can fix them yourself. I am not that good and my solution is to use package.mask liberally some times. One example is that python3.11 will not compile so I masked it and force all programs to use python3.10 instead. Also make sure to check that you are install stable branches and not testing branches. For some reason th chroot likes to install testing versions.

create your SD card and move the items over

Booting from a pi is different than amd64. I believe the pi4 is mostly hardcoded to boot form kernel8.img and it reads a config.txt and cmdline.txt for how to do the boot. An example I use is from Raspbian. If you wish to boot from another kernel, you will have to set kernel= in your config.txt

config.txt 

# For more options and information see
# http://rpf.io/configtxt
# Some settings may impact device functionality. See link above for details

# uncomment if you get no picture on HDMI for a default "safe" mode
#hdmi_safe=1

# uncomment the following to adjust overscan. Use positive numbers if console
# goes off screen, and negative if there is too much border
#overscan_left=16
#overscan_right=16
#overscan_top=16
#overscan_bottom=16

# uncomment to force a console size. By default it will be display's size minus
# overscan.
#framebuffer_width=1280
#framebuffer_height=720

# uncomment if hdmi display is not detected and composite is being output
#hdmi_force_hotplug=1

# uncomment to force a specific HDMI mode (this will force VGA)
#hdmi_group=1
#hdmi_mode=1

# uncomment to force a HDMI mode rather than DVI. This can make audio work in
# DMT (computer monitor) modes
#hdmi_drive=2

# uncomment to increase signal to HDMI, if you have interference, blanking, or
# no display
#config_hdmi_boost=4

# uncomment for composite PAL
#sdtv_mode=2

#uncomment to overclock the arm. 700 MHz is the default.
#arm_freq=800

# Uncomment some or all of these to enable the optional hardware interfaces
#dtparam=i2c_arm=on
#dtparam=i2s=on
#dtparam=spi=on

# Uncomment this to enable infrared communication.
#dtoverlay=gpio-ir,gpio_pin=17
#dtoverlay=gpio-ir-tx,gpio_pin=18

# Additional overlays and parameters are documented /boot/overlays/README

# Enable audio (loads snd_bcm2835)
dtparam=audio=on

# Automatically load overlays for detected cameras
camera_auto_detect=1

# Automatically load overlays for detected DSI displays
display_auto_detect=1

# Enable DRM VC4 V3D driver
dtoverlay=vc4-kms-v3d
max_framebuffers=2

# Run in 64-bit mode
arm_64bit=1

# Disable compensation for displays with overscan
disable_overscan=1

[cm4]
# Enable host mode on the 2711 built-in XHCI USB controller.
# This line should be removed if the legacy DWC2 controller is required
# (e.g. for USB device mode) or if USB support is not required.
otg_mode=1

[all]

[pi4]
# Run as fast as firmware / board allows
arm_boost=1

[all]

cmdline.txt 

dwc_otg.lpm_enable=0 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 cgroup_enable=memory elevator=deadline rootwait

Create your USB device you want and read what other people have done. There are optimizations and things like that can be important. 

#exit your chroot at this point after the stage4. You will be generating your pi at this point.

sudo mkdir /mnt/pi-mount
sudo mount /dev/sdb2 /mnt/pi-mount
sudo mv /usr/aarch64-unknown-linux-gnu/stage4.tar.gz /mnt/pi-mount/.
cd /mnt/pi-mount
sudo tar -xvf stage4.tar.gz --numeric-owner --xattrs-include='*.*'
sudo mount /dev/sda1 boot
sudo cp -r ~/opt/raspberry-pi-boot/* /mnt/pi-mount/boot/.
sudo cp -r ~/opt/modules/<Kernel-version> /mnt/pi-mount/usr/lib/modules/.

Now your boot and your modules and your root folder should be set up. Add the cmdline.txt and the config.txt to your mount boot.

This system should be bootable in its current state. Although it will only contain what you put in it for the stage4. Now to set up a binhost and change the pi make.conf.

Setting up binhost in your pi

At this point your pi will boot and you can proceed with everything inside the pi via ssh. The PI itself will need a new make.conf. The main point of this version is to finally remove all the qemu/crossdev from it and have it use packages.

My pi make.conf is show below. 

# Note: profile variables are set/overridden in profile/ files:
# etc/portage/profile/use.force (overrides kernel_* USE variables)
# etc/portage/profile/make.defaults (overrides ARCH, KERNEL, ELIBC variables)



CFLAGS="-O2 -pipe -fomit-frame-pointer -mcpu=cortex-a72 "

CXXFLAGS="${CFLAGS}"

MAKEOPTS="-j4"
ACCEPT_KEYWORDS="arm64"
ACCEPT_LICENSE="*"
EMERGE_DEFAULT_OPTS="--ask --verbose"
FEATURES="getbinpkg"
L10N="en en-UTF8"
PORTAGE_BINHOST="ssh://<your-username>@<Your-local-ip>/usr/aarch64-unknown-linux-gnu/var/cache/binpkgs/"
VIDEO_CARDS="v3d vc4 egl gles1 gles2"
INPUT_DEVICES="libinput evdev"




USE="X pulseaudio xorg  -acl -cups"

#EMERGE_DEFAULT_OPTS="$EMERGE_DEFAULT_OPTS --exclude app-emulation/qemu"

The main thing for this make.conf is to set up where the PORTAGE_BINHOST will look. The easiest for me was to just have it scp it over and use ssh-agent and ssh-add for pass wordless authentication. You can run all the heavy compiling in the QEMU on your AMD64 and just pull the packages you want via this way.

In general it should just work and no python workarounds are needed. On the PI it is now a normal gentoo operating system.

Wifi

To get wifi to work you need to install two packages for the RPI4. The following two packages are.

root #emerge --ask sys-boot/rasberrypi-firmware
root #emerge --ask sys-firmware/raspberrypi-wifi-ucode

Getting X working

X seems to have problems in crossdev for setting up the correct paths. In order to get X you may need to recompile all of the X related programs on the pi itself. It should still work with bin-hosting afterwards though. An additional item is needed though and that is to tell the pi to have graphics enabled. The following method can enable it in the config.txt

#Inside the config.txt in /boot do the following

#uncomment
#hdmi_drive=2
#dtoverlay=vc4-fkms-v3d

#add
hdmi_drive=1

The fastest and best way to compile but gets weird.

Crossdev without chroot is the best solution I can think of.

Crossdev has problems where it gets confused with your make.conf file of your real system and the crossdev make.conf. There is one solution and that is to build a VM or chroot env where you have only qemu/binfmt/crossdev inside of it. After the system is built and configured you can delete or rename your make.conf in your chroot/VM environment. This will let you use crossdev only in your system giving a MASSIVE speed up. Most packages will compile now unlike before but some will have problems. This gets weird in the sense that you may have a chroot inside of a chroot to do this.

However setting this up in some VM amd64 system following the system and deleting as a said would work the best and probably the most sane. Then you can set up a bridged network device in virt-manager and expose the VM to your network and binhost will work. So far this method has done me wonders as I have setup a binhost system via this method. Seems to have problems with vim/firefox though with crossdev.

Bridge eth0 with br0 to pass through with to virt-manager to expose VM to local network

Creating a VM seems to be a very good setup as a whole for doing this. You can "fix" a lot of the problems and still expose the device to the network.

To create your bridge do the following command. 

sudo ip link add name br0 type bridge
sudo ip link set dev br0 up
sudo ip link set dev eth0 master br0

Now pass this over in virt-manager and do all steps in the first part. Your VM will be the entire section you want to deal with and rename your make.conf to make.conf.bak and cross-dev should work much better.

Installing inside the VM

Since the install is to be as minimal as possible, only get what is needed to boot, dhcpcd, crossdev, and qemu. After you build the system, you need to remove the make.conf file from your host system inside the VM. 

mv /etc/portage/make.conf /etc/portage/make.conf.bak.bak
Retrieved from "https://wiki.gentoo.org/index.php?title=User:Rebeltrouper/Raspberry_Pi4_and_PI400_64_bit_install&oldid=1198729"