User:Fearedbliss/Installing Gentoo Linux On ZFS

Install Gentoo Linux on OpenZFS
Author: Jonathan Vasquez (fearedbliss) Contact: jon@xyinn.org

Preface
This guide will show you how to install Gentoo Linux on x86_64 with:

* UEFI-GPT (EFI System Partition - Unencrypted FAT32 partition as per UEFI Spec) * /boot on ZFS (Unencrypted) * /, /home on ZFS (Encrypted ZFS if desired) * swap on a regular partition * OpenZFS 0.8.4 * GRUB 2.04 * systemd * Gentoo Stable (x86_64)

Download the System Rescue CD + ZFS ISO
You will need to download my modified System Rescue CD that includes ZFS from my server.

Linux
We will be creating a UEFI Bootable USB since this guide will be showing you how to install Gentoo Linux on ZFS with UEFI Enabled.

For the following commands, we will assume that your USB is /dev/sdg.

Create the FAT32 Filesystem on the USB
We will now create the FAT32 filesystem on the USB. This needs to be FAT32 since this is the filesystem used in the UEFI Specification.

The label we will use for this partition will be in the following format SYSRCDXYZ, where XYZ is the version number of the System Rescue CD you downloaded.

For example, if you are using System Rescue CD 6.1.6, the label will be SYSRCD616.

Copy files over from ISO to USB
And that's it! You now have a Bootable UEFI USB.

Windows
Rufus is the USB Utility I recommend when on Windows. You can Download Rufus here.


 * 1) Start Rufus
 * 2) Select your USB Device from the Device drop down.
 * 3) Select your ISO by clicking SELECT.
 * 4) Partition Scheme: MBR
 * 5) Target system: BIOS or UEFI
 * 6) Volume label: SYSRCD613 (Change last 3 numbers for the version of your System Rescue CD ISO)
 * 7) File system: FAT32
 * 8) Cluster size: 4096 bytes (Default)
 * 9) Click START.

This should be all that's necessary to have a Bootable UEFI USB.

Assumptions

 * Only installing Gentoo on a single drive (Multiple drives in the same pool should automatically work).
 * The /boot pool is featureless and unencrypted.
 * The /boot/efi is an unencrypted FAT32 partition as per UEFI spec.
 * Your swap partition is outside of ZFS and on a dedicated partition.
 * GRUB 2.04+ is being used
 * My modified System Rescue CD + ZFS iso is being used.
 * bliss-initramfs is being used as your initramfs.
 * bliss-kernel is being used as your kernel.

You are free to substitute any of the above for whatever you want. However, support will only be provided from me when the above configuration is used. Also, this guide is the way I install Gentoo, not exactly the way the handbook has it.

Boot your system into the ISO
Since this is highly computer dependent, you will need to figure out how to boot your USB on your system and get to the live environment. You may need to disable Secure Boot if that causes your USB to be rejected. Make sure your system BIOS/UEFI is set up to boot UEFI devices, rather than BIOS devices (Legacy).

Confirm that you booted in UEFI Mode
After you booted into the Live CD, make sure that you booted into UEFI mode by typing the following:

If the above directory is empty or doesn't exist, you are not in UEFI mode. Reboot and boot into UEFI mode.

Partition
We will now partition the drive and aim to create the following layout:

/dev/sda1  | 512 MiB       |   EFI System Partition                     | /boot/efi /dev/sda2  | 1024 MiB      |   Boot Partition (ZFS, No Feature Flags)   | /boot /dev/sda3  | 2048 MiB      |   swap                                     | swap /dev/sda4  | Rest of Disk  |   ZFS (or Encrypted ZFS)                   | /, /home, ...

Open up your drive in GNU parted and tell it to use optimal alignment:

Create GPT partition layout
This will delete all partitions and create a new GPT table.

Final View
Exit the application

Create your zpool
Create your zpool which will contain your drives and datasets:

If you want to use native zfs encryption (with a passphrase), simply specify the -O encryption=on -O keyformat=passphrase options as well.

Create your zfs datasets
We will keep it simple and just create a few datasets for /, and /home. ZFS is extremely flexible and you can easily add or remove datasets in the future.

Create your boot pool
It is safer for us to create a separate zpool that has all feature flags disabled. This is because even though grub currently supports the latest feature flags for the 0.6.5 release of ZFS, new feature flags added to ZFS without proper bootloader support can make your system unbootable. Since GRUB 2 has solid support for a zpool with no feature flags, we will create a separate featureless boot pool. However, our main system will still have all features enabled.

Verify everything looks good
You can verify that all of these things worked by running the following:

Now we are ready to install Gentoo!

Set your date and time
Let's say it's August 24, 2014 @ 2:48 PM (will be 14:48 in 24 hour time), we would do the following:

Preparing to chroot
First let's mount our efi boot partition in our chroot directory:

and now download the systemd amd64 image here and extract it:

Edit fstab
Everything is on zfs so we don't need anything in here except for the efi directory and swap entries. My fstab looks as follows:

Modify make.conf
Let's modify our /etc/portage/make.conf so we can start installing stuff with a good base (Change it to what you need):

Get the portage tree
Copy the default example portage config

Using the 'bliss-kernel' configuration
If you don't want to use the pre-compiled binaries for bliss-kernel, you can use the bliss-kernel kernel configurations as a base for your kernel. The configuration is ready to go for general desktop, laptop, and server use, and it is also configured to work with ZFS. The bliss-kernel configuration is simply a slightly tweaked Fedora config.

You can download the bliss-kernel configurations here.

Disable 'initramfs' USE flag
Since we are using bliss-initramfs, we will disable the 'initramfs' USE flag on bliss-kernel-bin since that will pull in dracut and other dependencies.

Set kernel symlink to this kernel
After you finish installing the kernel, make sure to set your kernel symlink to the kernel that just got pulled in. This will allow us to build the ZFS modules for this kernel.

Find the kernel that just got installed

Once you found it, you can select it with its number

Install required applications
Unmask bliss-initramfs >= 9.0.0 (Provides new/easier configuration and native ZFS encryption support)

Unmask latest ZFS versions

Enable ZFS support in GRUB

Now install the apps:

Installing the bootloader onto your drive
We will need to install the bootloader onto the drive. Before we do that however, let's see if GRUB can detect our /boot and /boot/efi filesystem types:

This should say 'zfs'. If it doesn't, then something is wrong and your system will not boot!

This should say 'fat'. If it doesn't, then something is wrong and your system will not boot!

Before we install the bootloader, we will need to have read/write access to the efi nvram variables. Let's remount our efivars now:

Now run the following to install the bootloader to the drive:

The above command will install the grub bootloader files into /boot and the efi files into /boot/efi. It should return a "Installation finished. No error reported." message. If it doesn't, then something is wrong and your system will not boot!

Make the GRUB 2 configuration file
You can use the following configuration file as a basis for your system:

If you are using native ZFS encryption, add the encrypted flag to your kernel line as well.

Generating new zpool.cache file before/after reboot
ZFS is very sensitive about the data that is contained in the zpool.cache file and at this point, when we reboot, the information in it might not be completely accurate. To ensure we have a good cache file, we have instructed bliss-initramfs in the bootloader config above, to ignore the current cachefile on the system, and make a new one that is up-to-date. We only need to do this once.

Remove zpool.cache refresh flag from bootloader configuration
Open up your grub.cfg and remove the refresh flag from the kernel line.

Take a snapshot of your new system
Since we now have a working system, we will snapshot it in case we ever want to go back or recover files:

You can view the contents of these snapshots by checking their respective and hidden .zfs directories:

Limiting the ARC size
If you want to cap the ZFS ARC from growing past a certain point, you can put the number of bytes inside the /etc/modprobe.d/zfs.conf file, and then remake your initramfs. When the system starts up, and the module is loaded, these options will be passed to the zfs kernel module.

(Temporary) Change the ARC max for the running system to 4 GB

(Permanent) Save the 4 GB ARC cap as a loadable kernel parameter

Once we have the above file created, let's regenerate the initramfs. bliss-initramfs will automatically detect that this file exists and copy it into the initramfs. When you reboot your machine, the initramfs will load up the zfs kernel module with the parameters found in the file.

Recovery
If you need to get back into your zpool from a livecd environment, you'll simply need to import your pool again using the following commands:

If your pool is encrypted, you will also want to either pass the -l (lower case L) option your the zpool import command above (Which will ask you for the pool's passphrase and thus load the keys), or you can use the following command after you imported the pool without keys:

This will also ask you for the passphrase and load the key for the pool. Once you do that, you should be able to see the available value for your encrypted pool by doing:

You'll also probably want to import your boot pool and your efi partition as well:

Bliss ZFS Scripts
The following scripts allow you to automatically:


 * Take snapshots of your pool
 * Replicate the pool to another pool (Full and Incremental Backups)
 * Clean the old snapshots on your pools.

You can download, customize, and install the scripts into your /usr/local/sbin directory. Github.

And that's it. Enjoy!