User:Fearedbliss/Installing Gentoo Linux On ZFS

Install Gentoo Linux on OpenZFS
Author: Jonathan Vasquez (fearedbliss) Last Updated: 2022-10-13-2145

Preface
This guide will show you how to install Gentoo Linux on x86_64 with:

* UEFI-GPT (EFI System Partition - Unencrypted FAT32 partition as per UEFI Spec) * /boot on ZFS (Featureless & Unencrypted) * /, /home on ZFS (Encrypted ZFS if desired) * swap on a regular partition * OpenZFS 2.1.4 * GRUB 2.04+ * OpenRC (or systemd) * Gentoo Stable (x86_64)

Required Tools
You will need an ISO that contains OpenZFS. Luckily, the Gentoo Admin CD provides the needed packages. You can download the "Admin CD" from the Gentoo Downloads page. After that, we'll use it to make a bootable USB.

Linux
For the following commands, we will assume that your USB is /dev/sdg.

Windows
Rufus is the USB Utility I recommend when on Windows. You can Download Rufus here.


 * 1) Start Rufus
 * 2) Select your USB Device from the Device drop down.
 * 3) Select your ISO by clicking SELECT.
 * 4) Partition Scheme: MBR
 * 5) Target system: BIOS or UEFI
 * 6) Volume label: ADMINCD
 * 7) File system: FAT32
 * 8) Cluster size: 4096 bytes (Default)
 * 9) Click START.

This should be all that's necessary to have a bootable USB.

Assumptions

 * Only installing Gentoo on a single drive (Multiple drives in the same pool should automatically work).
 * The /boot pool is featureless and unencrypted.
 * The /boot/efi is an unencrypted FAT32 partition as per UEFI spec.
 * Your swap partition is outside of ZFS and on a dedicated partition.
 * GRUB 2.04+ is being used
 * Gentoo Admin CD (Contains OpenZFS)
 * Kernel: gentoo-kernel-bin
 * Initramfs: bliss-initramfs
 * Init: OpenRC

You are free to substitute any of the above for whatever you want. However, support will only be provided from me when the above configuration is used. Also, this guide is the way I install Gentoo, not exactly the way the handbook has it.

Boot your system into the ISO
Since this is highly computer dependent, you will need to figure out how to boot your USB on your system and get to the live environment. You may need to disable Secure Boot if that causes your USB to be rejected. Make sure your system BIOS/UEFI is set up to boot UEFI devices, rather than BIOS devices (Legacy).

Confirm that you booted in UEFI Mode
After you boot into the Live CD, make sure that you booted into UEFI mode by typing the following:

If the above directory is empty or doesn't exist, you are not in UEFI mode. Reboot and boot into UEFI mode.

Partition
We will now partition the drive and aim to create the following layout (for the rest of the guide we will assume /dev/nvme0n1 is your main drive):

/dev/nvme0n1p1  | 512 MiB       |   EFI System Partition                     | /boot/efi /dev/nvme0n1p2  | 1024 MiB      |   Boot Partition (ZFS, No Feature Flags)   | /boot /dev/nvme0n1p3  | 2048 MiB      |   swap                                     | swap /dev/nvme0n1p4  | Rest of Disk  |   ZFS (or Encrypted ZFS)                   | /, /home, ...

Open up your drive in GNU parted and tell it to use optimal alignment:

Create GPT partition layout
This will delete all partitions and create a new GPT table.

Final View
Exit the application

Create your zpool
Create your zpool which will contain your drives and datasets:

If you want to use native zfs encryption (with a passphrase), simply specify the -O encryption=on -O keyformat=passphrase options as well.

Create your zfs datasets
We will keep it simple and just create a few datasets for /, and /home. ZFS is extremely flexible and you can easily add or remove datasets in the future.

Create your boot pool
It is safer for us to create a separate zpool that has all feature flags disabled. This is because even though grub currently supports the latest feature flags for the 0.6.5 release of ZFS, new feature flags added to ZFS without proper bootloader support can make your system unbootable. Since GRUB 2 has solid support for a zpool with no feature flags, we will create a separate featureless boot pool. However, our main system will still have all features enabled.

Verify everything looks good
You can verify that all of these things worked by running the following:

Now we are ready to install Gentoo!

Set your date and time
Let's say it's June 18, 2022 @ 11:48 PM (will be 23:48 in 24 hour time), we would do the following:

Preparing to chroot
First let's mount our EFI boot partition in our chroot directory:

and download the OpenRC (or systemd) amd64 image here and extract it:

Edit fstab
Everything is on zfs so we don't need anything in here except for the efi directory and swap entries. My fstab looks as follows:

Modify make.conf
Let's modify our /etc/portage/make.conf so we can start installing stuff with a good base (Change it to what you need):

Get the portage tree
Copy the default example portage config

Kernel Installation
For simplicity, we will just use the prebuilt gentoo kernel.

Disable 'initramfs' USE flag
Since we are using bliss-initramfs, we will disable the 'initramfs' USE flag on gentoo-kernel-bin.

Install required applications
Enable ZFS support in GRUB

Now install the apps:

Installing the bootloader onto your drive
We will need to install the bootloader onto the drive. Before we do that however, let's see if GRUB can detect our /boot and /boot/efi filesystem types:

This should say 'zfs'. If it doesn't, then something is wrong and your system will not boot!

This should say 'fat'. If it doesn't, then something is wrong and your system will not boot!

Before we install the bootloader, we will need to have read/write access to the efi nvram variables. Let's remount our efivars now:

Now run the following to install the bootloader to the drive:

The above command will install the grub bootloader files into /boot and the efi files into /boot/efi. It should return a "Installation finished. No error reported." message. If it doesn't, then something is wrong and your system will not boot!

Make the GRUB 2 configuration file
You can use the following configuration file as a basis for your system:

If you are using native ZFS encryption, add the encrypted flag to your kernel line as well.

Generating new zpool.cache file before/after reboot
ZFS is very sensitive about the data that is contained in the zpool.cache file and at this point, when we reboot, the information in it might not be completely accurate. To ensure we have a good cache file, we have instructed bliss-initramfs in the bootloader config above, to ignore the current cachefile on the system, and make a new one that is up-to-date. We only need to do this once.

Generate the initramfs and move the file to its correct location
Before generating the initramfs, make sure to edit the /etc/bliss-initramfs/settings.json and include any kernel modules needed to boot your system. In this guide, we will need to load the nvme driver since gentoo-kernel-bin has this driver compiled as a module. If your system fails too boot, it may be related to a missing kernel module not being loaded. Follow the instructions in the recovery section for getting back into your environment.

Now generate your initramfs and move it to the correct location:

Final steps before reboot
OpenRC

systemd

And now for the magic commands (cross fingers lol):

Remove zpool.cache refresh flag from bootloader configuration
Open up your grub.cfg and remove the refresh flag from the kernel line.

Import your boot pool
You can mount your boot pool now so that your /boot directory will be available on subsequent reboots. The boot pool will be automatically remembered through the cachefile:

Take a snapshot of your new system
Since we now have a working system, we will snapshot it in case we ever want to go back or recover files:

You can view the contents of these snapshots by checking their respective and hidden .zfs directories:

Limiting the ARC size
If you want to cap the ZFS ARC from growing past a certain point, you can put the number of bytes inside the /etc/modprobe.d/zfs.conf file, and then remake your initramfs. When the system starts up, and the module is loaded, these options will be passed to the zfs kernel module.

(Temporary) Change the ARC max for the running system to 4 GB

(Permanent) Save the 4 GB ARC cap as a loadable kernel parameter

Once we have the above file created, let's regenerate the initramfs. bliss-initramfs will automatically detect that this file exists and copy it into the initramfs. When you reboot your machine, the initramfs will load up the zfs kernel module with the parameters found in the file.

Recovery
If you need to get back into your zpool from a livecd environment, you'll simply need to import your pool again using the following commands:

If your pool is encrypted, you will also want to either pass the -l (lower case L) option your the zpool import command above (Which will ask you for the pool's passphrase and thus load the keys), or you can use the following command after you imported the pool without keys:

This will also ask you for the passphrase and load the key for the pool. Once you do that, you should be able to see the available value for your encrypted pool by doing:

You'll also probably want to import your boot pool and your efi partition as well:

{{Note|If you have trouble booting your pool upon reboot, try adding the refresh flag during boot.}

Bliss ZFS Scripts
The following scripts allow you to automatically:


 * Take snapshots of your pool
 * Replicate the pool to another pool (Full and Incremental Backups)
 * Clean the old snapshots on your pools.

You can download, customize, and install the scripts into your /usr/local/sbin directory. Github.

And that's it. Enjoy!