User:Oishishou/Oishishou's guide to root on ZFS
This is a guide to building ZFS as a built-in kernel feature instead of a module, allowing for early ZFS support. It includes a walkthrough of initializing and mounting a ZFS dataset at the initramfs stage, allowing for root to be a ZFS dataset.
Initial assumptions
This guide assumes the user is already familiar and comfortable with manually configuring and building the kernel and a basic initramfs with Busybox, as well as ZFS. Other resources exist in this wiki to better understand these processes at this time. A full example of a basic initramfs is included at the end.
The kernel sources used herein are from sys-kernel/gentoo-sources, and are linked to /usr/src/linux, which will be the working directory used for the kernel.
The initramfs files will be located at /usr/src/initramfs.
This guide is presented as two major parts: Patching ZFS into the kernel, and preparing an initramfs for root on a ZFS dataset.
Environment
Work must be done in an environment already supporting ZFS. This is typically a live boot environment with ZFS modules loaded, such as the Admin CD or LiveGUI USB image available on the downloads page.
Patching the kernel sources
Preparing the kernel sources
Kernel sources need to be configured and built before the ebuild can be configured against them. Ensure modules are enabled before building. Modules may be disabled or left enabled after the patch is complete, depending on user preference and unrelated needs.
[*] Enable loadable module support Search for <code>CONFIG_MODULES</code> to find this item.
From /usr/src/linux:
root #make allPreparing the ebuild
Portage configuration
The ZFS package does not support patching the kernel by default, and must have a USE flag enabled, which is masked by default.
Unmasking USE flag
/etc/portage/package.use.maskUnmasking USE flag in package.use.masksys-fs/zfs -kernel-builtin
Setting USE flag
/etc/portage/package.useSetting USE flag in package.usesys-fs/zfs kernel-builtin
Configuring ebuild
Once the kernel is built with module support, the ebuild can be configured. ZFS version 2.3.2 is used here.
root #ebuild /var/db/repos/gentoo/sys-fs/zfs-kmod/zfs-kmod-2.3.2.ebuild clean configureCopying the ZFS source code to kernel source directory
Once configured, the kernel sources can be patched. Ensure the correct ZFS version is being used. Version 2.3.2 is used here.
root #(cd /var/tmp/portage/sys-fs/zfs-kmod-2.3.2/work/zfs-kmod-2.3.2/ && ./copy-builtin /usr/src/linux)The kernel sources are now patched.
Cleaning up
To clean up the kernel sources so they are in a fresh state, but with the patch applied:
root #make mrproperThis will also delete any .config files in the sources, so they should be copied to a location outside of /usr/src/linux to preserve, unless they are not needed (e.g. starting fresh, using existing .config stored elsewhere).
Configuring the kernel
Once the kernel is patched, all that needs to be done is configuring ZFS in the kernel config.
File systems --->
<*> ZFS filesystem support Search for <code>CONFIG_ZFS</code> to find this item.
Alternatively, if an existing .config file is being used which already has ZFS enabled, it can simply be copied into the kernel source directory like any manual kernel upgrade.
The kernel is now ready with ZFS support. It cannot directly import or mount zpool or ZFS datasets, though. Userspace binaries are still required for this.
Installing userspace binaries
It is important to ensure you have up-to-date binaries to actually use ZFS functions.
root #emerge --ask sys-fs/zfsBuilding the initramfs
Assumed initramfs
The initramfs example used here is a basic hand-built initramfs using busybox sh via statically linked Busybox. For further details on creating initramfs, see various pages listed at initramfs.
The working directory for the initramfs is /usr/src/initramfs.
Userspace binaries
Unfortunately, zfs and zpool cannot be statically linked, and thus require all dynamically linked libraries associated. This can be achieved via the ldd command (part of sys-libs/glibc) or the lddtree (part of app-misc/pax-utils. lddtree is used in this guide due to conveniently being able to copy all necessary files with one command.
Installing lddtree
The python USE flag is necessary for copying all files with one command.
/etc/portage/package.useSetting USE flag in package.useapp-misc/pax-utils python
root #emerge --ask app-misc/pax-utilsPreparing initramfs with ZFS utilities
Use lddtree to copy zfs and zpool, along with necessary libraries.
root #lddtree /usr/bin/{zfs,zpool} --copy-to-tree /usr/src/initramfsVerify files have been copied properly:
This example does not represent the files or directory structure needed for an initrams, only the effects of lddtree --copy-to-tree.
The necessary libraries can potentially change. Do not assume this will always be the necessary files. Always use lddtree in a fresh directory to ensure proper files. It will create the necessary directory structure.
root #ls -R /usr/src/initramfs/usr/src/initramfs: lib64 usr /usr/src/initramfs/lib64: ld-linux-x86-64.so.2 /usr/src/initramfs/usr: bin lib64 /usr/src/initramfs/usr/bin: zfs zpool /usr/src/initramfs/usr/lib64: libblkid.so.1 libcrypto.so.3 libm.so.6 libtirpc.so.3 libuuid.so.1 libzfs_core.so.3 libz.so.1 libcap.so.2 libc.so.6 libnvpair.so.3 libudev.so.1 libuutil.so.3 libzfs.so.6
Example initramfs
This is an example of a very basic initramfs, including basic device nodes and init script. Block device nodes will vary depending on your system and other needs.
This example uses nvme0n1 as the only block device, zroot0 as the zpool, and zroot0/systemroot as the dataset with the root filesystem.
Example directory structure and files
root #ls -R /usr/src/initramfs/usr/src/initramfs: bin dev etc init lib lib64 mnt opt proc root run sbin sys usr var /usr/src/initramfs/bin: busybox /usr/src/initramfs/dev: console null nvme0n1 zero /usr/src/initramfs/etc: /usr/src/initramfs/lib: /usr/src/initramfs/lib64: ld-linux-x86-64.so.2 /usr/src/initramfs/mnt: systemroot /usr/src/initramfs/mnt/systemroot: /usr/src/initramfs/opt: /usr/src/initramfs/proc: /usr/src/initramfs/root: /usr/src/initramfs/run: /usr/src/initramfs/sbin: /usr/src/initramfs/sys: /usr/src/initramfs/usr: bin lib lib64 sbin /usr/src/initramfs/usr/bin: zfs zpool /usr/src/initramfs/usr/lib: /usr/src/initramfs/usr/lib64: libblkid.so.1 libcrypto.so.3 libm.so.6 libtirpc.so.3 libuuid.so.1 libzfs_core.so.3 libz.so.1 libcap.so.2 libc.so.6 libnvpair.so.3 libudev.so.1 libuutil.so.3 libzfs.so.6 /usr/src/initramfs/usr/sbin: /usr/src/initramfs/var:
Test the initramfs files by chrooting into the initramfs directory with chroot /usr/src/initramfs busybox sh and attempt to run the zfs and zpool commands for function. They will fail if libraries are missing.
Example init scripts
There are two ways to mount a ZFS dataset, depending on dataset properties: Standard ZFS mounting using the zfs command, and "legacy" mounting using the mount command. In both cases, the zpool command is necessary to import the zpool.
Due to the complexity involved in standard ZFS mounting, this example uses the "legacy" option. This requires the dataset property mountpoint=legacy to be set. It requires the mount command instead of zfs mount, as well as allows for fstab entries to mount the dataset. mount must use -t zfs and -o zfsutil to ensure the mount command uses the zfs command added to the initramfs. These options are also required in any fstab entries on the actual root filesystem.
/usr/src/initramfs/initinitrams with legacy mounting#!/bin/busybox sh
clear
mount -t proc none /proc
mount -t sysfs none /sys
mount -t devtmpfs none /dev
zpool import -d /dev/mapper/zroot0 zroot0
mount -t zfs -o zfsutil zroot0/systemroot /mnt/systemroot
umount /proc
mount --move /sys /mnt/systemroot/sys
mount --move /dev /mnt/systemroot/dev
exec switch_root /mnt/systemroot /sbin/init
Normally, it is preferable to unmount /proc, /sys, and /dev, allowing them to be mounted by the actual init system, as opposed to the initramfs. However, importing the zpool causes /sys and /dev to be continuously in use, so the existing mounted virtual filesystems must remain active by moving them into the actual root filesystem via mount --move.