User:Fearedbliss/Installing Gentoo Linux On ZFS

From Gentoo Wiki
Jump to: navigation, search

Contents

Install Gentoo Linux on OpenZFS

Author: Jonathan Vasquez (fearedbliss)
Contact: jon@xyinn.org

Preface

This guide will show you how to install Gentoo Linux on AMD64 with:

 * UEFI-GPT (EFI System Partition) - This will be on a FAT32 unencrypted partition as per UEFI Spec.
 * /, /home, and /boot on ZFS
 * swap on regular partition
 * OpenZFS 0.8.3
 * GRUB 2.04 (Bootloader)
 * systemd
 * Gentoo Stable (amd64)

Notes on other stuff you can do in the ZFS install can be found here.

Required Tools

Download the System Rescue CD + ZFS ISO

You will need to download my modified System Rescue CD that includes ZFS from one of the mirrors on the front page.

Linux

We will be creating a UEFI Bootable USB since this guide will be showing you how to install Gentoo Linux on ZFS with UEFI Enabled.

For the following commands, we will assume that your USB is /dev/sdg.

Format the USB
root #parted -a optimal /dev/sdg
(parted)unit mib
(parted)mklabel msdos
(parted)mkpart primary 1 -1
(parted)set 1 boot on
(parted)print
                                                            
Model: ATA VBOX HARDDISK (scsi)
Disk /dev/sdg: 8192MiB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start    End      Size     Type     File system  Flags
 1      1.00MiB  8191MiB  8190MiB  primary               boot, lba
(parted)quit
Create the FAT32 Filesystem on the USB

We will now create the FAT32 filesystem on the USB. This needs to be FAT32 since this is the filesystem used in the UEFI Specification. The label we will use for this partition will be in the following format SYSRCDXYZ, where XYZ is the version number of the System Rescue CD you downloaded.

For example, if you are using System Rescue CD 6.0.7, the label will be SYSRCD607.

root #mkfs.fat -F32 -n SYSRCD607 /dev/sdg1
Make folders to mount USB and ISO
root #mkdir /tmp/usb
root #mkdir /tmp/sysresccd
Mount your USB and your ISO
root #mount -o loop,ro sysresccd-6.0.7_zfs_0.8.3.iso /tmp/sysresccd
root #mount /dev/sdg1 /tmp/usb
Copy files over from ISO to USB
root #rsync -avP /tmp/sysresccd/ /tmp/usb/

And that's it! You now have a Bootable UEFI USB.

Windows

Rufus is the USB Utility I recommend when on Windows. You can Download Rufus here.

  1. Start Rufus
  2. Select your USB Device from the Device drop down.
  3. Select your ISO by clicking SELECT.
  4. Partition Scheme: MBR
  5. Target system: BIOS or UEFI
  6. Volume label: SYSRCD607 (Change last 3 numbers for the version of your System Rescue CD ISO)
  7. File system: FAT32
  8. Cluster size: 4096 bytes (Default)
  9. Click START.


This should be all that's necessary to have a Bootable UEFI USB.

Assumptions

  • Only installing Gentoo on one drive called /dev/sda (or /dev/nvme0n1, etc)
  • My modified System Rescue CD + ZFS iso is being used.
  • bliss-initramfs is being used as your initramfs.
  • bliss-kernel is being used as your kernel.


You are free to substitute any of the above for whatever you want if you know what you are doing (alternate kernels, initramfs generators, partition layouts (RAIDZ2?), etc). However, support will only be provided from me when the above configuration is used. Also, this guide is the way I install Gentoo, not exactly the way the handbook has it.

Boot your system into the ISO

Since this is highly computer dependent, you will need to figure out how to boot your USB on your system and get to the live environment. You may need to disable Secure Boot if that causes your USB to be rejected. Make sure your system BIOS/UEFI is set up to boot UEFI devices, rather than BIOS devices (Legacy).

Confirm that you booted in UEFI Mode

After you booted into the Live CD, make sure that you booted into UEFI mode by typing the following:

root #ls /sys/firmware/efi

If the above directory is empty or doesn't exist, you are not in UEFI mode. Reboot and boot into UEFI mode.

Warning
Continuing the installation without being in UEFI mode will most likely yield an unbootable system. If you want to install in BIOS mode, you will need a different setup.

Partition

We will now partition the drive and aim to create the following layout:

/dev/sda1   | 512 MiB       |   EFI System Partition                | /boot/efi
/dev/sda2   | 1024 MiB      |   Boot Partition (No Feature Flags)   | /boot
/dev/sda3   | 2048 MiB      |   swap                                | swap
/dev/sda4   | Rest of Disk  |   ZFS                                 | /, /home, ...
Warning
There are many UEFI motherboard firmwares that are extremely buggy. We will attempt to use a 512 MiB FAT32 partition configuration to increase success.

Open up your drive in GNU parted and tell it to use optimal alignment:

root #parted -a optimal /dev/sda
Warning
Keep in mind that all of the following operations will affect the disk immediately. GNU parted does not stage changes like fdisk or gdisk.

Set the units to Mebibytes

(parted)unit mib

Create GPT partition layout

This will delete all partitions and create a new GPT table.

(parted)mklabel gpt

Create and label your partitions

(parted)mkpart primary 1 513
(parted)mkpart primary 513 1013
(parted)mkpart primary 1013 3585
(parted)mkpart primary 3585 -1
(parted)name 1 esp
(parted)name 2 boot
(parted)name 3 swap
(parted)name 4 rootfs

Set the bootable flag on the ESP partition

(parted)set 1 boot on

Final View

(parted)print
Model: ATA VBOX HARDDISK (scsi)
Disk /dev/sda: 476940MiB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
  
Number   Start      End        Size     File system  Name   Flags
 1       1.00MiB    513MiB     512MiB                esp    boot, esp
 2       513MiB     1537MiB    1024MiB               boot
 3       1537MiB    3585MiB    2048MiB               swap
 4       3585MiB    476939MiB  473354MiB             rootfs

Exit the application

(parted)quit

Format your drives

Create your boot partition

root #mkfs.fat -F32 /dev/sda1
Warning
This partition needs to be FAT32 due to it being an UEFI requirement. If it isn't, your system will not boot!

Create your zpool

Create your zpool which will contain your drives and datasets:

root #zpool create -f -o ashift=12 -o cachefile= -O compression=lz4 -O acltype=posixacl -O atime=off -O xattr=sa -O normalization=formD -m none -R /mnt/gentoo tank /dev/sda4

Create your zfs datasets

We will keep it simple and just create a few datasets for /, and /home. ZFS is extremely flexible and you can easily add or remove datasets in the future.

root #zfs create tank/gentoo
root #zfs create -o mountpoint=/ tank/gentoo/os
root #zfs create -o mountpoint=/home tank/gentoo/home

Create your boot pool

It is safer for us to create a separate zpool that has all feature flags disabled. This is because even though grub currently supports the latest feature flags for the 0.6.5 release of ZFS, new feature flags added to ZFS without proper bootloader support can make your system unbootable. Since GRUB 2 has solid support for a zpool with no feature flags, we will create a separate featureless boot pool. However, our main system will still have all features enabled.

root #zpool create -f -d -o ashift=12 -o cachefile= -m /boot -R /mnt/gentoo boot /dev/sda2
Warning
Do not do a 'zpool upgrade' on your boot pool. Doing this will mostly likely leave your system unbootable.

Create your swap

root #mkswap -f /dev/sda3
root #swapon /dev/sda3
Note
  • Do not put your swap inside the zvol. I've experienced lockups when my RAM is 100% and the system starts swapping while the swap is on ZFS. It doesn't crash when the swap is on a normal partition.

Verify everything looks good

You can verify that all of these things worked by running the following:

root #zpool status
  pool: boot
 state: ONLINE
status: The pool is formatted using a legacy on-disk format.  The pool can
        still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
        pool will no longer be accessible on software that does not support
        feature flags.
  scan: none requested
config: 

        NAME        STATE     READ WRITE CKSUM
        boot        ONLINE       0     0     0
          sda2      ONLINE       0     0     0

errors: No known data errors

  pool: tank
 state: ONLINE
  scan: none requested
config: 

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          sda3      ONLINE       0     0     0

errors: No known data errors
root #zfs list
NAME                USED  AVAIL  REFER  MOUNTPOINT
boot                      36.2M  742M     35.1M  /boot
tank                       315G  402G      192K  none
tank/gentoo                310G  402G      192K  none
tank/gentoo/home           320K  402G      320K  /home
tank/gentoo/os             310G  402G      310G  /

Now we are ready to install Gentoo!

Installing Gentoo

Set your date and time

Let's say it's August 24, 2014 @ 2:48 PM (will be 14:48 in 24 hour time), we would do the following:

root #date 082414482014
Note
08 24 1448 2014 (Month, Date, 24 Hour Time, Year). It will be UTC at the moment but you can fix it once you reboot.

Preparing to chroot

First let's mount our efi boot partition in our chroot directory:

root #cd /mnt/gentoo
root #mkdir boot/efi
root #mount /dev/sda1 boot/efi

and now download the systemd amd64 image here and extract it:

root #wget <file>
root #tar xpf <file>

Copy zpool cache

root #mkdir etc/zfs
root #cp /etc/zfs/zpool.cache etc/zfs

Copy network settings

root #cp /etc/resolv.conf etc/

Mount required devices

root #mount --rbind /dev dev && mount --make-rslave dev
root #mount --rbind /proc proc && mount --make-rslave proc
root #mount --rbind /sys sys && mount --make-rslave sys

Chroot into your environment

root #env -i HOME=/root TERM=$TERM chroot . bash -l

Inside the chroot

Edit fstab

Everything is on zfs so we don't need anything in here except for the boot and swap zvol entries. My fstab looks as follows:

root #nano /etc/fstab

/dev/sda1               /boot/efi       vfat            noauto        1 2
/dev/sda3               none            swap            sw            0 0
Warning
Do not auto-mount the /boot/efi partition. If you do, the init system will create the /boot/efi folder if it doesn't exist, and that will prevent the boot zpool from mounting.

Modify make.conf

Let's modify our /etc/portage/make.conf so we can start installing stuff with a good base (Change it to what you need):

root #nano /etc/portage/make.conf

USE="-branding"

# This should be your number of processors + 1
MAKEOPTS="-j5"

EMERGE_DEFAULT_OPTS="--with-bdeps=y --keep-going=y"
FEATURES="buildpkg"
LINGUAS="en en_US"

# This is required so that when we compile GRUB later, EFI support is built.
GRUB_PLATFORMS="efi-64"

Get the portage tree

Copy the default example portage config

root #cp /usr/share/portage/config/repos.conf /etc/portage/repos.conf
root #emerge --sync

Kernel Configuration

Using the 'bliss-kernel' configuration

If you don't want to use the pre-compiled binaries for 'bliss-kernel', you can use the 'bliss-kernel' kernel configurations as a base for your kernel. The configuration is ready to go for general desktop, laptop, and server use, and it is also configured to work with ZFS. The 'bliss-kernel' configuration is simply a slightly tweaked Fedora config.

You can download the 'bliss-kernel' configurations here.

Using the pre-compiled 'bliss-kernel' binaries

Install the kernel

root #emerge bliss-kernel-bin
Note

Consider pinning the kernel version that was installed so that when we do a future kernel upgrade, an emerge --depclean won't delete our kernel files from the /boot directory. If the bliss-kernel ebuild that emerge pulled to install was: bliss-kernel-bin-4.1.8, we would do:

root #emerge --noreplace =bliss-kernel-bin-4.1.8

Install required applications

Unmask latest ZFS versions

root #echo "sys-fs/zfs" >> /etc/portage/package.accept_keywords
root #echo "sys-fs/zfs-kmod" >> /etc/portage/package.accept_keywords

Enable ZFS support in GRUB

root #echo "sys-boot/grub libzfs" >> /etc/portage/package.use

Now install the apps:

root #emerge bliss-initramfs grub zfs

Installing the bootloader onto your drive

We will need to install the bootloader onto the drive. Before we do that however, let's see if GRUB can detect our /boot and /boot/efi filesystem types:

root #grub-probe /boot

This should say 'zfs'. If it doesn't, then something is wrong and your system will not boot!

root #grub-probe /boot/efi

This should say 'fat'. If it doesn't, then something is wrong and your system will not boot!

Before we install the bootloader, we will need to have read/write access to the efi nvram variables. Let's remount our efivars now:

root #mount -o remount,rw /sys/firmware/efi/efivars/

Now run the following to install the bootloader to the drive:

root #grub-install --efi-directory=/boot/efi

The above command will install the grub bootloader files into /boot and the efi files into /boot/efi. It should return a "Installation finished. No error reported." message. If it doesn't, then something is wrong and your system will not boot!

Warning
If after a reboot your UEFI motherboard doesn't detect your bootloader, it might be because of a buggy motherboard firmware. Try to pass the "--removable" flag to grub install and try again. This will install the efi file to the EFI specification's fallback directory.

Make the GRUB 2 configuration file

You can use the following configuration file as a basis for your system:

root #nano /boot/grub/grub.cfg

set default=0
set timeout=1

insmod part_gpt
insmod fat
insmod efi_gop
insmod efi_uga

menuentry "Gentoo - 4.1.8-FC.01" {
    linux /@/vmlinuz-4.1.8-FC.01 root=tank/gentoo/os by=id elevator=noop quiet logo.nologo triggers=zfs refresh
    initrd /@/initrd-4.1.8-FC.01
}

Generating new zpool.cache file before/after reboot

ZFS is very sensitive about the data that is contained in the zpool.cache file and at this point, when we reboot, the information in it might not be completely accurate. To ensure we have a good cache file, we have instructed bliss-initramfs in the bootloader config above, to ignore the current cachefile on the system, and make a new one that is up-to-date. We only need to do this once.

Generate the initramfs and move the file to its correct location

root #bliss-initramfs zfs 4.1.8-FC.01
root #mv initrd-4.1.8-FC.01 /boot

Final steps before reboot

root #systemctl enable zfs.target
root #systemctl enable zfs-import-cache
root #systemctl enable zfs-mount
root #systemctl enable zfs-import.target
root #passwd
root #exit
root #reboot

After you reboot

Remove zpool.cache refresh flag from bootloader configuration

Open up your grub.cfg and remove the 'refresh' flag from the kernel line.

root #nano /boot/grub/grub.cfg

Take a snapshot of your new system

Since we now have a working system, we will snapshot it in case we ever want to go back or recover files:

root #zfs snapshot boot@2016-03-25-0000-01-INSTALL
root #zfs snapshot tank/gentoo/os@2016-03-25-0000-01-INSTALL
root #zfs snapshot tank/gentoo/home@2016-03-25-0000-01-INSTALL

You can view the contents of these snapshots by checking their respective and hidden .zfs directories:

root #ls /boot/.zfs/snapshots
root #ls /.zfs/snapshots
root #ls /home/.zfs/snapshots
Note
We are naming the snapshots in this format because it's the format that my 'bliss-zfs-scripts' use, and it is also very readable.

Limiting the ARC size

If you want to cap the ZFS ARC from growing past a certain point, you can put the number of bytes inside the /etc/modprobe.d/zfs.conf file, and then remake your initramfs. When the system starts up, and the module is loaded, these options will be passed to the zfs kernel module.

(Temporary) Change the ARC max for the running system to 4 GB

root #echo 4294967296 >> /sys/module/zfs/parameters/zfs_arc_max

(Permanent) Save the 4 GB ARC cap as a loadable kernel parameter

root #echo "options zfs zfs_arc_max=4294967296" >> /etc/modprobe.d/zfs.conf

Once we have the above file created, let's regenerate the initramfs. bliss-initramfs will automatically detect that this file exists and copy it into the initramfs. When you reboot your machine, the initramfs will load up the zfs kernel module with the parameters found in the file.

root #bliss-initramfs zfs 4.1.8-FC.01
root #mv initrd-4.1.8-FC.01 /boot

Other stuff you might want to install

Bliss ZFS Scripts

The following scripts allow you to automatically:

  • Take snapshots of your pool
  • Replicate the pool to another pool (Full and Incremental Backups)
  • Clean the old snapshots on your pools.

You can download, customize, and install the scripts into your /usr/local/sbin directory. Github.

And that's it. Enjoy!