User:NeddySeagoon/Pi4 Root in LVM

From Gentoo Wiki
Jump to:navigation Jump to:search
This page is a Work in Progress, so its a bit sketchy


Root in LVM is almost conventional. The path described here is for a kernel with everything built in and an initrd that contains only userspace tools, so that its like firmware. That means that its kernel agnostic.

The binaries for the initrd will be installed in their own space, so that they can be reused in years to come if the initrd init script needs to be changed.

The initrd will be compiled using the kernel provided usr/gen_init tool that is compiled with the kernel.

this is built for the compile host, not the target. That matters if the kernel is normally cross compiled

The Test Environment

The Pi4 used to develop and test this process normally has /boot on the SD card, root on a partition on SSD over USB and everything else in LVM on the SSD.

This arrangement makes it possible to boot the Pi normally, then swap the SD card out for the SD card containing the test environment. Recovery after a test boot failure is then reduced to swapping the SDs back.


Root in LVM on a Pi can be made to work but its a bit hacky. The main issue is that block devices are not instantly available, so that the vgchange command will fail as the underlying block device is not available when it runs.

The usual rootwait or rootdelay won't help as they wait for a block device and its the LV underlying block device, which isn't known, that needs to be waited for.

dm-init, the way to have root in LVM without an initrd will not work for this reason.


The foundation kernel sources
The foundation firmware to go in /boot
A system to build the initrd binaries on
A system to build the kernel on
Cross compiling should work but doing everything on the Pi will be safer
A Pi with /boot on SD card and everything else on USB allows the SD card to be swapped

Prepare The SD Card

The partitioning, making filesystems, putting a stage3 into the LVM root and the Foundation firmware files into /boot need not be done on a Pi.


Make two partitions, one for boot the other for LVM.


Make at least one LV for root. Untar a stage3 to root. Others can be added later.

Prepare /boot

For a Pi4 the completed boot should look like

bcm2711-rpi-4-b.dtb        fixup4db.dat                    kernel-5.10.27-v8+
cmdline.txt                fixup4x.dat                     LICENCE.broadcom
cmdline.txt_LVM_no_initrd  foundation_bcm2711-rpi-4-b.dtb  overlays
config.txt                 foundation_kernel8.img          start4cd.elf
COPYING.linux              foundation_overlays             start4db.elf
fixup4cd.dat               initramfs_rootinLVM_1           start4.elf
fixup4.dat                 initrd                          start4x.elf
The foundation_* files are not requited

Be sure that kernel-5.10.27-v8+, overlays/ and bcm2711-rpi-4-b.dtb are a matching set from the kernel build.

Build the Kernel

Fetch the foundation kernel sources and do the usual host configuration and build. At the time time of writing, 5.10.y is the branch to use.

During configuration, check that everything required to mount root is built in. Using the default configuration is a good start an the Pi normally operates this way. Do check the device mapper support.

Install the kernel Image file, overlays and the *.dtb to /boot

Install the kernel modules to the SD card LVM root.

Prepare The Initrd

To keep all the bits together, make a directory /root/initrd to house init initramfs_list and initrd_bins/ The initramfs_list describes what goes where in the initrd. init is the init script that will control getting root in LVM mounted and initrd_bins/ the binaries that will go into the initrd.

It is possible to use the installed system binaries too

Populating /root/initrd/initrd_bins/

We need busybox, lvm2 and the real mount command as busybox mount does not support mount by UUID. We make everything statically linked, as far as possible, turn off all the documentation, turn off all environment provided USE flags then turm off all the ebuild IUSE +USE preferences we don't want.

root #FEATURES="noman nodoc noinfo" USE="-* static static-libs -cramfs -logger -ncurses -nls -pam -python readline -suid -udev" emerge --root="/root/initrd/initrd_bins" busybox lvm2 -DNav

That command builds the things we need, all the run time dependencies and installs into /root/initrd/initrd_bins.

Build time dependencies, installed in the build system root, may need to be rebuilt with changed USE flags

The initramfs_list

This file is in the format required by the usr/gen_init command. It has a help option.

# directory structure
dir /proc       755 0 0
dir /usr        755 0 0
dir /bin        755 0 0
dir /sys        755 0 0
dir /var        755 0 0
dir /lib        755 0 0
dir /lib64      755 0 0
dir /sbin       755 0 0
dir /mnt        755 0 0
dir /mnt/root   755 0 0
dir /etc        755 0 0
dir /root       700 0 0
dir /dev        755 0 0
dir /dev/mapper 755 0 0

# As we are going to use devtmpfs, we need to do this *after* devtmpfs is mounted.
#slink /dev/stderr                      /proc/self/fd/2                 777 0 0
#slink /dev/stdin                       /proc/self/fd/0                 777 0 0
#slink /dev/stdout                      /proc/self/fd/1                 777 0 0

# busybox is not supposed to need this any more but its harmless.
slink /bin/cat                          /bin/busybox                    777 0 0
slink /bin/cut                          /bin/busybox                    777 0 0
slink /bin/ln                           /bin/busybox                    777 0 0
slink /bin/ls                           /bin/busybox                    777 0 0
# We need the real mount command.
#slink /bin/mount                       /bin/busybox                    777 0 0
slink /bin/umount                       /bin/busybox                    777 0 0
slink /bin/sleep                        /bin/busybox                    777 0 0
slink /bin/uname                        /bin/busybox                    777 0 0
slink /bin/findfs                       /bin/busybox                    777 0 0
slink /sbin/switch_root                 /bin/busybox                    777 0 0
slink /etc/mtab                         /proc/self/mounts               777 0 0

# mount looks in /lib but is in /lib64
slink /lib/        /lib64/    777 0 0

# Moved out of the init script
# Does lvm still build static - yes.
slink /sbin/vgchange                    /sbin/lvm.static                777 0 0

# Now the programs that will ho all the work
# busybox static.
file /bin/busybox               /root/initrd/initrd_bins/bin/busybox            755 0 0
file    /sbin/lvm.static        /root/initrd/initrd_bins/sbin/lvm.static        755 0 0 
file    /bin/mount              /root/initrd/initrd_bins/bin/mount              755 0 0  

# We need to use the real mount command as bb does not do mount by UUID
# The dynamic linker and the bits of mount.
file    /lib64/    /root/initrd/initrd_bins/lib64/    755 0 0
file    /lib64/            /root/initrd/initrd_bins/lib64/            755 0 0
file    /lib64/            /root/initrd/initrd_bins/lib64/            755 0 0
file    /lib64/                /root/initrd/initrd_bins/lib64/                755 0 0

# our init script
file    /init                           /root/initrd/init               755 0 0

The init Script

This does the work and if it fails, tells us why and gives us a busybox shell, so we can investigate.

This Works but needs some fine tuning - maybe the real mount command is not needed
#!/bin/busybox sh

rescue_shell() {
    echo "$@"
    echo "Something went wrong. Dropping you to a shell."
# have time to read the message
    /bin/sleep 20
    /bin/busybox --install -s
    exec /bin/sh

# allow the use of UUIDs or filesystem lables
uuidlabel_root() {
    for cmd in $(cat /proc/cmdline) ; do
        case $cmd in
            type=$(echo $cmd | cut -d= -f2)
echo $type
            echo "Mounting rootfs"
            if [ $type == "LABEL" ] || [ $type == "UUID" ] ; then
                uuid=$(echo $cmd | cut -d= -f3)
                mount -o ro $(findfs "$type"="$uuid") /mnt/root
                mount -o ro $(echo $cmd | cut -d= -f2) /mnt/root

# start for real here
# temporarily mount proc and sys

echo "Starting initramfs init script"

export PATH

#echo "Mount proc"
mount -t proc proc /proc || rescue_shell "/proc failed to mount"
#busybox sleep 20

#echo "Mount sysfs"
mount -t sysfs sysfs /sys  || rescue_shell "/sys failed to mount"

# We have devtmpfs so use it.
mount -t devtmpfs dev /dev  || rescue_shell "/dev failed to mount"

# After /dev is mounted or the /dev targets will be hidden.
ln -s /proc/self/fd/2 /dev/stderr || rescue_shell "symlink /dev/stderr failed"
ln -s /proc/self/fd/0 /dev/stdin || rescue_shell "symlink /dev/stdin failed"
ln -s /proc/self/fd/1 /dev/stdout || rescue_shell "symlink /dev/stdout failed"

# disable kernel messages from popping onto the screen
###echo 0 > /proc/sys/kernel/printk
# clear the screen

# mmc cards are slow and detected asynchronously we have
# root in LVM on mmc. rootwait and rootdelay cannot be used as root is not a device 
sleep 2
# start the Pi_4 volume group
# This is on the SD card for testing but likely to be moved to USB.
vgchange -ay Pi4 || rescue_shell "lvchange Pi_4 failed"

# mounting rootfs on /mnt/root
uuidlabel_root || rescue_shell "Error with uuidlabel_root"

# clean up. The init process will remount proc sys and dev later
umount /proc
umount /sys
umount /dev

echo "All done. Switching to real root."

# switch to the real root and execute init
exec /sbin/switch_root /mnt/root /sbin/init

# Should never get here, but we may as well look round rather
# than panic trying to kill init. 
rescue_shell "Oops, fell off the end of the init script"

Putting All Initrd Bits Together

Remember to mount /boot if its not already mounted.

root #cd /usr/src/linux
root #usr/gen_init_cpio /root/initrd/initramfs_list > /boot/initramfs_rootinLVM_1


The /boot/config.txt file needs this fragment to boot.

# Booting  Stuff

# as we don't use default kernel names, we must set 64 bit mode and name the kernel
# No '=' after initramfs below
initramfs initramfs_rootinLVM_1 followkernel 

# The .dtb is free with the kernel. You did use the one to match the kernel
# Likewise the .dtbo that go in /boot/overlays.

The older entries


are no longer required but will be used if provided.


It must be a single line
dwc_otg.lpm_enable=0 8250.nr_uarts=1 console=ttyAMA0,115200  console=tty cma=256M@256M root=UUID=ac1556c0-1c8d-4a1a-82f9-cd6cfbad903d rootfstype=ext4
Use your own root=UUID=, not mine, as shown above
root #blkid

will show the filesystem UUIDs.

Other Niceties

The test environment is a stage3, plus a kernel, the initrd being tested and the foundation GPU code. Its about as minimal as it can get and still be Gentoo.

In no particular order, set the following ...


The first sign that it worked.

Console Keymap

You need to be able to log in as root to shutdown cleanly

Root Password

Its good to shut down cleanly if if works.