User:NeddySeagoon/Pi4 Root in LVM
This page is a Work in Progress, so its a bit sketchy
Overview
Root in LVM is almost conventional. The path described here is for a kernel with everything built in and an initrd that contains only userspace tools, so that its like firmware. That means that its kernel agnostic.
The binaries for the initrd will be installed in their own space, so that they can be reused in years to come if the initrd init script needs to be changed.
The initrd will be compiled using the kernel provided usr/gen_init tool that is compiled with the kernel.
this is built for the compile host, not the target. That matters if the kernel is normally cross compiled
The Test Environment
The Pi4 used to develop and test this process normally has /boot on the SD card, root on a partition on SSD over USB and everything else in LVM on the SSD.
This arrangement makes it possible to boot the Pi normally, then swap the SD card out for the SD card containing the test environment. Recovery after a test boot failure is then reduced to swapping the SDs back.
Drawbacks
Root in LVM on a Pi can be made to work but its a bit hacky. The main issue is that block devices are not instantly available, so that the vgchange command will fail as the underlying block device is not available when it runs.
The usual rootwait or rootdelay won't help as they wait for a block device and its the LV underlying block device, which isn't known, that needs to be waited for.
dm-init, the way to have root in LVM without an initrd will not work for this reason.
Prerequisites
The foundation kernel sources The foundation firmware to go in /boot A system to build the initrd binaries on A system to build the kernel on
Cross compiling should work but doing everything on the Pi will be safer
A Pi with /boot on SD card and everything else on USB allows the SD card to be swapped
Prepare The SD Card
The partitioning, making filesystems, putting a stage3 into the LVM root and the Foundation firmware files into /boot need not be done on a Pi.
Partitioning
Make two partitions, one for boot the other for LVM.
Filesystems
Make at least one LV for root. Untar a stage3 to root. Others can be added later.
Prepare /boot
For a Pi4 the completed boot should look like
bcm2711-rpi-4-b.dtb fixup4db.dat kernel-5.10.27-v8+ cmdline.txt fixup4x.dat LICENCE.broadcom cmdline.txt_LVM_no_initrd foundation_bcm2711-rpi-4-b.dtb overlays config.txt foundation_kernel8.img start4cd.elf COPYING.linux foundation_overlays start4db.elf fixup4cd.dat initramfs_rootinLVM_1 start4.elf fixup4.dat initrd start4x.elf
The foundation_* files are not requited
Be sure that kernel-5.10.27-v8+, overlays/ and bcm2711-rpi-4-b.dtb are a matching set from the kernel build.
Build the Kernel
Fetch the foundation kernel sources and do the usual host configuration and build. At the time time of writing, 5.10.y is the branch to use.
During configuration, check that everything required to mount root is built in. Using the default configuration is a good start an the Pi normally operates this way. Do check the device mapper support.
Install the kernel Image file, overlays and the *.dtb to /boot
Install the kernel modules to the SD card LVM root.
Prepare The Initrd
To keep all the bits together, make a directory /root/initrd to house init initramfs_list and initrd_bins/ The initramfs_list describes what goes where in the initrd. init is the init script that will control getting root in LVM mounted and initrd_bins/ the binaries that will go into the initrd.
It is possible to use the installed system binaries too
Populating /root/initrd/initrd_bins/
We need busybox, lvm2 and the real mount command as busybox mount does not support mount by UUID. We make everything statically linked, as far as possible, turn off all the documentation, turn off all environment provided USE flags then turm off all the ebuild IUSE +USE preferences we don't want.
root #FEATURES="noman nodoc noinfo" USE="-* static static-libs -cramfs -logger -ncurses -nls -pam -python readline -suid -udev" emerge --root="/root/initrd/initrd_bins" busybox lvm2 -DNavThat command builds the things we need, all the run time dependencies and installs into /root/initrd/initrd_bins.
Build time dependencies, installed in the build system root, may need to be rebuilt with changed USE flags
The initramfs_list
This file is in the format required by the usr/gen_init command. It has a help option.
# directory structure dir /proc 755 0 0 dir /usr 755 0 0 dir /bin 755 0 0 dir /sys 755 0 0 dir /var 755 0 0 dir /lib 755 0 0 dir /lib64 755 0 0 dir /sbin 755 0 0 dir /mnt 755 0 0 dir /mnt/root 755 0 0 dir /etc 755 0 0 dir /root 700 0 0 dir /dev 755 0 0 dir /dev/mapper 755 0 0 # As we are going to use devtmpfs, we need to do this *after* devtmpfs is mounted. #slink /dev/stderr /proc/self/fd/2 777 0 0 #slink /dev/stdin /proc/self/fd/0 777 0 0 #slink /dev/stdout /proc/self/fd/1 777 0 0 # busybox is not supposed to need this any more but its harmless. slink /bin/cat /bin/busybox 777 0 0 slink /bin/cut /bin/busybox 777 0 0 slink /bin/ln /bin/busybox 777 0 0 slink /bin/ls /bin/busybox 777 0 0 # We need the real mount command. #slink /bin/mount /bin/busybox 777 0 0 slink /bin/umount /bin/busybox 777 0 0 slink /bin/sleep /bin/busybox 777 0 0 slink /bin/uname /bin/busybox 777 0 0 slink /bin/findfs /bin/busybox 777 0 0 slink /sbin/switch_root /bin/busybox 777 0 0 slink /etc/mtab /proc/self/mounts 777 0 0 # mount looks in /lib but ld-linux-aarch64.so.1 is in /lib64 slink /lib/ld-linux-aarch64.so.1 /lib64/ld-linux-aarch64.so.1 777 0 0 # Moved out of the init script # Does lvm still build static - yes. slink /sbin/vgchange /sbin/lvm.static 777 0 0 # Now the programs that will ho all the work # busybox static. file /bin/busybox /root/initrd/initrd_bins/bin/busybox 755 0 0 file /sbin/lvm.static /root/initrd/initrd_bins/sbin/lvm.static 755 0 0 file /bin/mount /root/initrd/initrd_bins/bin/mount 755 0 0 # We need to use the real mount command as bb does not do mount by UUID # The dynamic linker and the bits of mount. file /lib64/ld-linux-aarch64.so.1 /root/initrd/initrd_bins/lib64/ld-linux-aarch64.so.1 755 0 0 file /lib64/libmount.so.1 /root/initrd/initrd_bins/lib64/libmount.so.1 755 0 0 file /lib64/libblkid.so.1 /root/initrd/initrd_bins/lib64/libblkid.so.1 755 0 0 file /lib64/libc.so.6 /root/initrd/initrd_bins/lib64/libc.so.6 755 0 0 # our init script file /init /root/initrd/init 755 0 0
The init Script
This does the work and if it fails, tells us why and gives us a busybox shell, so we can investigate.
This Works but needs some fine tuning - maybe the real mount command is not needed
#!/bin/busybox sh
rescue_shell() {
echo "$@"
echo "Something went wrong. Dropping you to a shell."
# have time to read the message
/bin/sleep 20
/bin/busybox --install -s
exec /bin/sh
}
# allow the use of UUIDs or filesystem lables
uuidlabel_root() {
for cmd in $(cat /proc/cmdline) ; do
case $cmd in
root=*)
type=$(echo $cmd | cut -d= -f2)
echo $type
echo "Mounting rootfs"
if [ $type == "LABEL" ] || [ $type == "UUID" ] ; then
uuid=$(echo $cmd | cut -d= -f3)
mount -o ro $(findfs "$type"="$uuid") /mnt/root
else
mount -o ro $(echo $cmd | cut -d= -f2) /mnt/root
fi
;;
esac
done
}
# start for real here
# temporarily mount proc and sys
echo "Starting initramfs init script"
PATH=/sbin:/bin
export PATH
#echo "Mount proc"
mount -t proc proc /proc || rescue_shell "/proc failed to mount"
#busybox sleep 20
#echo "Mount sysfs"
mount -t sysfs sysfs /sys || rescue_shell "/sys failed to mount"
# We have devtmpfs so use it.
mount -t devtmpfs dev /dev || rescue_shell "/dev failed to mount"
# After /dev is mounted or the /dev targets will be hidden.
ln -s /proc/self/fd/2 /dev/stderr || rescue_shell "symlink /dev/stderr failed"
ln -s /proc/self/fd/0 /dev/stdin || rescue_shell "symlink /dev/stdin failed"
ln -s /proc/self/fd/1 /dev/stdout || rescue_shell "symlink /dev/stdout failed"
# disable kernel messages from popping onto the screen
###echo 0 > /proc/sys/kernel/printk
# clear the screen
###clear
# mmc cards are slow and detected asynchronously we have
# root in LVM on mmc. rootwait and rootdelay cannot be used as root is not a device
sleep 2
# start the Pi_4 volume group
# This is on the SD card for testing but likely to be moved to USB.
vgchange -ay Pi4 || rescue_shell "lvchange Pi_4 failed"
# mounting rootfs on /mnt/root
uuidlabel_root || rescue_shell "Error with uuidlabel_root"
# clean up. The init process will remount proc sys and dev later
umount /proc
umount /sys
umount /dev
echo "All done. Switching to real root."
# switch to the real root and execute init
exec /sbin/switch_root /mnt/root /sbin/init
# Should never get here, but we may as well look round rather
# than panic trying to kill init.
rescue_shell "Oops, fell off the end of the init script"
Putting All Initrd Bits Together
Remember to mount /boot if its not already mounted.
root #cd /usr/src/linuxroot #usr/gen_init_cpio /root/initrd/initramfs_list > /boot/initramfs_rootinLVM_1/boot/config.txt
The /boot/config.txt file needs this fragment to boot.
# Booting Stuff # as we don't use default kernel names, we must set 64 bit mode and name the kernel arm_64bit=1 kernel=kernel-5.10.27-v8+ # No '=' after initramfs below initramfs initramfs_rootinLVM_1 followkernel # The .dtb is free with the kernel. You did use the one to match the kernel # Likewise the .dtbo that go in /boot/overlays. device_tree=bcm2711-rpi-4-b.dtb
The older entries
enable_gic=1 armstub=armstub8-gic.bin
are no longer required but will be used if provided.
/boot/cmdline.txt
It must be a single line
dwc_otg.lpm_enable=0 8250.nr_uarts=1 console=ttyAMA0,115200 console=tty cma=256M@256M root=UUID=ac1556c0-1c8d-4a1a-82f9-cd6cfbad903d rootfstype=ext4
Use your own root=UUID=, not mine, as shown above
root #blkidwill show the filesystem UUIDs.
Other Niceties
The test environment is a stage3, plus a kernel, the initrd being tested and the foundation GPU code. Its about as minimal as it can get and still be Gentoo.
In no particular order, set the following ...
Hostname
The first sign that it worked.
Console Keymap
You need to be able to log in as root to shutdown cleanly
Root Password
Its good to shut down cleanly if if works.