Btrfs/Native System Root Guide

From Gentoo Wiki
< Btrfs
Jump to: navigation, search
Article status
This article needs wikification.

This exercise is an update to the original example for re-basing a Gentoo installation's root filesystem to use btrfs found here. In this case, the existing system is a mirror set using two 2TB drives at /dev/sda and /dev/sdb. Two fresh 2TB drives have been added at /dev/sdc and /dev/sdd. In the original exercise, the mdadm array was kept to mirror the /boot partitions on the two drives, while the rest of the partitions were converted to btrfs subvolumes.

This second exercise explores the use of GRUB2 to fully convert the mirror set to use the multi-volume functionality of btrfs, implementing /boot as a mirrored btrfs filesystem and forgoing the need to use an initramfs file on /boot to provide early userspace mounting of the root subvolume. We also will be using the gptfdisk ebuild tools to make a GPT based partition table on the new mirror set in place of a legacy MBR table.

One of the lessons learned here is that GRUB2 (2.00_p5107-r2 here) and the current kernel (3.10.25 here) are not quite up to the job of directly booting a filesystem in a btrfs subvolume, at least, not one in a multi-disc set. However it is able to find a simple filesystem on the default volume of a multi-disc set.

Partitioning

Emerge gptfdisk if you don't already have it. It provides the gdisk, sgdisk, and cgdisk utilities for manipulating GPT partitions. These correspond to the legacy fdisk, sfdisk, and cfdisk utilities for MBR tables.

We followed the write up on GRUB2 here to put together the following GPT scheme that GRUB2 can use and got it working after a false start or two.

root #cgdisk
                                          cgdisk 0.8.6

                                      Disk Drive: /dev/sdc
                                   Size: 3907029168, 1.8 TiB

Part. #     Size        Partition Type            Partition Name
----------------------------------------------------------------
            1.8 TiB     free space

    [ Align  ]  [ Backup ]  [  Help  ]  [  Load  ]  [  New   ]  [  Quit  ]  [ Verify ]  [ Write  ]

(Choosing new)
First sector (40-3907029134, default = 40): 2048
Size in sectors or {KMGTP} (default = 3907027087): 6144
Hex code or GUID (L to show codes, Enter = 8300): EF02
Enter new partition name, or <Enter> to use the current name:
grub2biosboot

(resulting display)

                                          cgdisk 0.8.6

                                      Disk Drive: /dev/sdc
                                   Size: 3907029168, 1.8 TiB

Part. #     Size        Partition Type            Partition Name
----------------------------------------------------------------
            1007.0 KiB  free space
   1        3.0 MiB     BIOS boot partition       grub2biosboot
            1.8 TiB     free space

    [ Align  ]  [ Backup ]  [  Help  ]  [  Load  ]  [  New   ]  [  Quit  ]  [ Verify ]  [ Write  ]

(arrowing to highlight the 1.8TiB freespace entry and choosing New)


First sector (8192-3907029134, default = 8192):
Size in sectors or {KMGTP} (default = 3907020943): 500M
Hex code or GUID (L to show codes, Enter = 8300): 
Enter new partition name, or <Enter> to use the current name:
boot

(resulting display)

                                          cgdisk 0.8.6

                                      Disk Drive: /dev/sdc
                                   Size: 3907029168, 1.8 TiB

Part. #     Size        Partition Type            Partition Name
----------------------------------------------------------------
            1007.0 KiB  free space
   1        3.0 MiB     BIOS boot partition       grub2biosboot
   2        500.0 MiB   Linux filesystem          boot
            1.8 TiB     free space

    [ Align  ]  [ Backup ]  [  Help  ]  [  Load  ]  [  New   ]  [  Quit  ]  [ Verify ]  [ Write  ]


(arrowing to highlight the 1.8TiB freespace entry again and choosing New)

First sector (1032192-3907029134, default = 1032192): 
Size in sectors or {KMGTP} (default = 3905996943):
Hex code or GUID (L to show codes, Enter = 8300): 
Enter new partition name, or <Enter> to use the current name:
root


(resulting display)

                                          cgdisk 0.8.6

                                      Disk Drive: /dev/sdc
                                   Size: 3907029168, 1.8 TiB

Part. #     Size        Partition Type            Partition Name
----------------------------------------------------------------
            1007.0 KiB  free space
   1        3.0 MiB     BIOS boot partition       grub2biosboot
   2        500.0 MiB   Linux filesystem          boot
   3        1.8 TiB     Linux filesystem          root

    [ Align  ]  [ Backup ]  [  Help  ]  [  Load  ]  [  New   ]  [  Quit  ]  [ Verify ]  [ Write  ]

The grub2biosboot partition was what got missed on the first round and thus got added later.

  • 1007.0KiB free space - Will eventually get the boot record and leave enough of a gap for GRUB2 to park its BIOS.
  • biosboot - The partition type 0xEF02 (BIOS boot partition) must be set in order for GRUB2 to find it and use it. Some web pages suggest using EF00 (EFI System), but this will not work with the current version of GRUB2. The bare minimum for this size is 1mb, but some pages suggest using at least 2 MBs. We err on the side of caution and future bloat.
  • boot - GRUB2 will take about 32mb more of /boot than you may be used to so the usual suggestion of 200mb in days past is now more like 300mb - 500mb depending on how many kernels and initramfs filesystems you like to keep around. Set the partition type to the default which is 0x8300 for Linux.
  • root - We took the default for size to allocate the rest of the drive to the root partition. Once again the type is set to 0x8300.

The resulting table looks like the following in gdisk:

root #gdisk /dev/sdc
GPT fdisk (gdisk) version 0.8.6

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.

Command (? for help): p
Disk /dev/sdc: 3907029168 sectors, 1.8 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): A5C43D4A-ED5A-4173-BDC0-5D632C0BAEEF
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 3907029134
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048            8191   3.0 MiB     EF02  grub2biosboot
   2            8192         1032191   500.0 MiB   8300  boot
   3         1032192      3907029134   1.8 TiB     8300  root

Command (? for help): 
Note
You may not see the "MBR: protective" indication in the output unless an MBR has been written to the drive at least once.

Repeat the same partitioning on /dev/sdd. Similar to sfdisk, the sgdisk utility has the ability to dump the GPT table to a file and then reload that. However the output is binary, so we just went ahead and used cgdisk to create the same layout on /dev/sdd.

Filesystem creation

Since this is a two disk simple mirror, we specify raid1 for both metadata and data when making the two filesystems. If you have more than two drives, the current stable versions of kernel (3.10.25) and btrfs-progs (3.12-r1) now also make raid5 and raid6 available for options alongside raid1 and raid10.

root #mkfs -t btrfs -L BOOT -m raid1 -d raid1 /dev/sdc2 /dev/sdd2
SMALL VOLUME: forcing mixed metadata/data groups

WARNING! - Btrfs v3.12 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using

Turning ON incompat feature 'mixed-bg': mixed data and metadata block groups
Turning ON incompat feature 'extref': increased hardlink limit per file to 65536
Created a data/metadata chunk of size 8388608
adding device /dev/sdd2 id 2
fs created label BTBOOT on /dev/sdc2
        nodesize 4096 leafsize 4096 sectorsize 4096 size 1000.00MiB
Btrfs v3.12

root #mkfs -t btrfs -L BTROOT -m raid1 -d raid1 /dev/sdc3 /dev/sdd3

WARNING! - Btrfs v3.12 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using

Turning ON incompat feature 'extref': increased hardlink limit per file to 65536
adding device /dev/sdd3 id 2
fs created label BTROOT on /dev/sdc3
        nodesize 16384 leafsize 16384 sectorsize 4096 size 3.64TiB

Root Volume

We mount the default volume for the root partition on /mnt/newmirror but will be putting the actual contents into subvolumes with different btrfs features enabled or disabled.

root #mkdir /mnt/newmirror
root #mount -t btrfs -o defaults,noatime,compress=lzo,autodefrag /dev/sdc3 /mnt/newmirror


The new root filesystem will go onto a subvolume (activeroot) which is created on the mirror and then mounted to /mnt/newroot

root #mkdir /mnt/newroot
root #btrfs subvol create /mnt/newmirror/activeroot
root #mount -t btrfs -o defaults,noatime,compress=lzo,autodefrag,subvol=activeroot /dev/sdc3 /mnt/newroot

We mount the existing root filesystem to /mnt/rawroot and use tar to transfer things over while avoiding the dynamic stuff.

root #mkdir /mnt/rawroot
root #mount --bind / /mnt/rawroot
root #cd /mnt/rawroot
|tar cvpf - . | (cd /mnt/newroot; tar xpf -)|output=

40 or so gb worth of stuff in the output log goes by
}}

Boot volume

At this point /mnt/newroot/boot and the mountpoints for the other subvolumes we will be creating are already in place from the tar. We keep the mount options simple for /mnt/newroot/boot.

root #mount -t btrfs -o defaults,noatime /dev/sdc1 /mnt/newroot/boot
root #cd /boot
root #tar cvpf - . | (cd /mnt/newroot/boot; tar xpf -)

short and sweet output log goes by

Other volumes

/home is an obvious candidate for a subvolume, but they are easy to create and manage so you will probably have others. In our example we have the following:

  • /home
  • /distfiles - It doesn't do any good to enable compression strategies for a directory which just has compressed tarballs for the most part.
  • /vm - Keeping your virtual machine store in a separate volume eases snapshotting and migrations. We will enable compression here. At various points in btrfs history the use of autodefrag had impacts and issues on VM performance.
  • /vmcrypt - If the VM uses drive encryption, the whole compression strategy gets blown out of the water.
root #btrfs subvol create /mnt/newmirror/home
root #mount -t btrfs -o defaults,noatime,compress=lzo,autodefrag,subvol=home /dev/sdc3 /mnt/newroot/home
root #btrfs subvol create /mnt/newmirror/distfiles
root #mount -t btrfs -o defaults,noatime,autodefrag,subvol=distfiles /dev/sdc3 /mnt/newroot/distfiles
root #btrfs subvol create /mnt/newmirror/vm
root #mount -t btrfs -o defaults,noatime,compress=lzo,autodefrag,subvol=vm /dev/sdc3 /mnt/newroot/vm
root #btrfs subvol create /mnt/newmirror/vmcrypt
root #mount -t btrfs -o defaults,noatime,autodefrag,subvol=vmcrypt /dev/sdc3 /mnt/newroot/vmcrypt

This process takes place overnight and a good bit of the next day, so we will gloss over it

Chrooting into /mnt/newroot

We will chroot into the new root filesystem for the next set of steps:

  • Edit the fstab
  • Update the kernel to use an embedded initram filesystem
  • Install grub2 in place of grub0.97

Pre-chroot preparation

We do the usual prelims to allow GRUB to find things when installing.

root #mount --bind /dev /mnt/newroot/dev
root #mount -t proc none /mnt/newroot/proc


Edit your mtab to look something like this:

FILE /mnt/newroot/etc/mtabthe new root and boot as they will look to grub
/dev/sdc3 / btrfs rw,noatime,compress=lzo,autodefrag,subvol=activeroot 0 0
/dev/sdc2 /boot btrfs rw,noatime 0 0
root #chroot /mnt/newroot /bin/bash
root #env-update
>>> Regenerating /etc/ld.so.cache...
root #source /etc/profile

Updating fstab

We edit our new fstab to make it look something like this:

FILE /etc/fstabedited in the chroot so is actually /mnt/newroot/etc/fstab
# /etc/fstab: static file system information.
#
# noatime turns off atimes for increased performance (atimes normally aren't 
# needed; notail increases performance of ReiserFS (at the expense of storage 
# efficiency).  It's safe to drop the noatime options if you want and to 
# switch between notail / tail freely.
#
# The root filesystem should have a pass number of either 0 or 1.
# All other filesystems should have a pass number of 0 or greater than 1.
#
# See the manpage fstab(5) for more information.
#

# <fs>			<mountpoint>	<type>		<opts>		<dump/pass>
/dev/sr0		/mnt/cdrom	auto		ro,noauto,users	0 0

# glibc 2.2 and above expects tmpfs to be mounted at /dev/shm for 
# POSIX shared memory (shm_open, shm_unlink).
# (tmpfs is a dynamically expandable/shrinkable ramdisk, and will
#  use almost no memory if not populated with files)
shm			/dev/shm	tmpfs		nodev,nosuid,noexec	0 0
>>> Regenerating /etc/ld.so.cache...

LABEL=BTROOT	/mnt/btrfsmirror	btrfs	defaults,noatime,compress=lzo,autodefrag			0 0
LABEL=BTROOT	/			btrfs	defaults,noatime,compress=lzo,autodefrag,subvol=activeroot	0 0
LABEL=BTROOT	/home			btrfs	defaults,noatime,compress=lzo,autodefrag,subvol=home		0 0
LABEL=BTROOT	/distfiles		btrfs	defaults,noatime,autodefrag,subvol=distfiles			0 0
LABEL=BTROOT	/vm			btrfs	defaults,noatime,compress=lzo,autodefrag,subvol=vm		0 0
LABEL=BTROOT	/vmcrypt		btrfs	defaults,noatime,autodefrag,subvol=vmcrypt			0 0

LABEL=BTBOOT	/boot			btrfs	defaults,noatime						0 0

Embedding an initram filesystem

As we noted in the introduction, the current kernel and GRUB2 combination appear to work fine at least when searching for and mounting the simple /boot mirror btrfs filesystem in the default volume. However we put our new root in a subvolume in order to take advantage of snapshotting and rollback of root as necessary. To do that, we will follow the guide for Early Userspace Mounting again but will also draw upon one of its references in order to have the kernel build process make the archive and then embed it in the resulting bzImage file. This lets us make grub2_mkconfig do all of the heavy lifting without having to do a custom stanza in /etc/grub.d/40_custom.

root #cd /usr/src/linux
root #chmod +x usr/gen_init_cpio scripts/gen_initramfs_list.sh

Run menuconfig to use devtmpfs and to specify an initramfs_list that will be used to put an initramfs into the bzImage:

KERNEL
General setup  --->
    [*] Initial RAM filesystem and RAM disk (initramfs/initrd) support
    (/usr/src/linux/initramfs/initramfs_list) Initramfs source file(s)
Device Drivers  --->
    Generic Driver Options  --->
        -*- Maintain a devtmpfs filesystem to mount at /dev
        [*]  Automount devtmpfs at /dev, after the kernel mounted the rootfs
root #cd initramfs

initramfs_list

We don't want to bloat the bzImage too much so only add the essential busybox, filesystem tools and the nano editor. Use ldd to figure out any dependencies of additional commands that you might add.

FILE /usr/src/linux/initramfs/initramfs_listFile and attributes list to be used by kernel when doing the embed
# directory structure
dir /proc       755 0 0
dir /usr        755 0 0
dir /bin        755 0 0
dir /sys        755 0 0
dir /var        755 0 0
dir /lib64      755 0 0
dir /sbin       755 0 0
dir /mnt        755 0 0
dir /mnt/root   755 0 0
dir /mnt/boot	755 0 0
dir /etc        755 0 0
dir /root       700 0 0
dir /dev        755 0 0

# busybox
file /bin/busybox		/bin/busybox		755 0 0

#
# fsck deps
#
file /sbin/fsck			/sbin/fsck		755 0 0
file /lib64/libmount.so.1	/lib64/libmount.so.1	755 0 0
file /lib64/libblkid.so.1	/lib64/libblkid.so.1	755 0 0
file /lib64/libc.so.6		/lib64/libc.so.6	755 0 0
file /lib64/libuuid.so.1	/lib64/libuuid.so.1	755 0 0
file /lib64/ld-linux-x86-64.so.2  /lib64/ld-linux-x86-64.so.2 755 0 0

#
#  fsck.ext3 and added deps
#
file /sbin/fsck.ext3		/sbin/fsck.ext3		755 0 0
file /lib64/libext2fs.so.2	/lib64/libext2fs.so.2	755 0 0
file /lib64/libcom_err.so.2	/lib64/libcom_err.so.2	755 0 0
file /lib64/libe2p.so.2		/lib64/libe2p.so.2	755 0 0
file /lib64/libpthread.so.0	/lib64/libpthread.so.0	755 0 0

#
# btrfs utils and added deps
#
file /sbin/btrfs		/sbin/btrfs		755 0 0
file /sbin/btrfs-convert	/sbin/btrfs-convert	755 0 0
file /sbin/btrfs-debug-tree	/sbin/btrfs-debug-tree	755 0 0
file /sbin/btrfs-find-root	/sbin/btrfs-find-root	755 0 0
file /sbin/btrfs-image		/sbin/btrfs-image	755 0 0
file /sbin/btrfs-map-logical	/sbin/btrfs-map-logical	755 0 0
file /sbin/btrfs-show-super	/sbin/btrfs-show-super	755 0 0
file /sbin/btrfs-zero-log	/sbin/btrfs-zero-log	755 0 0
file /sbin/btrfsck		/sbin/btrfsck		755 0 0
file /sbin/btrfstune		/sbin/btrfstune		755 0 0
file /sbin/mkfs.btrfs		/sbin/mkfs.btrfs	755 0 0
file /lib64/libz.so.1		/lib64/libz.so.1	755 0 0
file /lib64/liblzo2.so.2	/usr/lib64/liblzo2.so.2 755 0 0

#
# Nano editor and added deps
#
file /sbin/nano			/usr/bin/nano		755 0 0
file /lib64/libncursesw.so.5	/lib64/libncursesw.so.5	755 0 0
file /lib64/libdl.so.2		/lib64/libdl.so.2	755 0 0

#
#  mknod and added deps
#
file /sbin/mknod		/bin/mknod		755 0 0

#
#  more viewer
#
file /sbin/more			/bin/more		755 0 0

#
#  init script
#
file    /init                   /usr/src/linux/initramfs/init             755 0 0

#
#  fstab
#
file	/etc/fstab		/usr/src/linux/initramfs/fstab	644 0 0

init script alternatives

We started with the init script from the previous exercise but have modified it to create two alternatives. The first should parse the /proc/cmdline to pull the information needed to mount root. It thus doesn't need to use the embedded fstab, but you can place a copy of your normal one in the initramfs directory in case you are dumped to the rescue shell.

The second is a basic mount script that uses an embedded fstab to explicitly spell out the mount entries for root and boot. We resorted to using this when we ran into problems with the init stage and then found that grub2 was not letting busybox open the console device.

/proc/cmdline parsing init script

This has not yet been shown to work during an actual boot, but the logic was tested against busybox on a running system.

FILE /usr/src/linux/initramfs/init
#!/bin/busybox sh

rescue_shell() {
    echo "$@"
    echo "Something went wrong. Dropping you to a shell."
    busybox --install -s
    exec /bin/sh
}

get_opt() {
	echo "$@" | cut -d "=" -f 2
}

mount_root() {
    ro=""
    root=""
    rootflags=""
    rootfstype="btrfs"

    #
    # This scan is probably not necessary, especially if the grub2 stanzas have put
    # enough info in /proc/cmdline to find root
    #
    echo "Scanning for btrfs filesystems..."
    /sbin/btrfs device scan
    /sbin/btrfs fi show

    for i in $(cat /proc/cmdline); do
	case $i in
	    root\=*)
		root=${i:5}
		;;
	    ro)
		ro="ro"
		;;
	    rootflags\=*)
		rootflags=${i:10}
		;;
	    rootfstype\=*)
		rootfstype=$(get_opt $i)
		;;
	esac
    done

    case $root in
	UUID\=*)
	    root="-U ${root:5}"
	    ;;

	LABEL\=*)
	    root="-L ${root:6}"
	    ;;
    esac

    if [[ ${ro} == "ro" ]] ; then
	if [[ ${rootflags} != "" ]] ; then
	    rootflags="ro,${rootflags}"
	else
	    rootflags="ro"
	fi
    fi
	
    echo "mounting /mnt/root"
    mount -t ${rootfstype} -o ${rootflags} ${root} /mnt/root
}

#
# temporarily mount proc and sys
#
mount -t proc none /proc
mount -t sysfs none /sys
mount -t devtmpfs none /dev

#
# disable kernel messages from popping onto the screen
#
echo 0 > /proc/sys/kernel/printk

#
# clear the screen
#
clear

#
# mount rootfs on /mnt/root
#
mount_root || rescue_shell "Error mounting root with /proc/cmdline=$(cat /proc/cmdline)"

#
#  If you are curious about the environment in effect at root, uncomment out these.
#  The rootfs must not be mounted readonly though.
#
#cat /proc/mounts >/mnt/root/root/mounts.txt
#ls -l / >/mnt/root/root/lsroot.txt
#ls -l /dev >/mnt/root/root/lsdev.txt
#env >/mnt/root/root/env.txt

echo "All done. Switching to real root."

#
# clean up. The init process will remount proc sys and dev later
#
umount /proc
umount /sys
umount /dev

#
# switch to the real root and execute init
#
exec switch_root /mnt/root /sbin/init


Basic mount init script

This basic fstab based mount version and the accompanying fstab appears to work:

FILE /usr/src/linux/initramfs/init
#!/bin/busybox sh

rescue_shell() {
    echo "$@"
    echo "Something went wrong. Dropping you to a shell."
    busybox --install -s
    exec /bin/sh
}

mount_root() {
    echo "scanning for btrfs filesystems.... will take about 5-10 seconds"
#
#  earlier versions of btrfs required devices on scan.  As of around 0.19.11 or so 
#  this now is a syntax error.  Also specifying the device= options in the fstab for
#  the root subvolume probably make this unnecessary now
#
#    /sbin/btrfs device scan /dev/sda /dev/sdb
    /sbin/btrfs device scan
    echo "mounting /mnt/root"
    mount /mnt/root
}

# temporarily mount proc and sys
mount -t proc none /proc
mount -t sysfs none /sys
mount -t devtmpfs none /dev

# disable kernel messages from popping onto the screen
echo 0 > /proc/sys/kernel/printk

# clear the screen
clear

# mounting rootfs on /mnt/root
mount_root || rescue_shell "Error with uuidlabel_root"

cat /proc/mounts >/mnt/root/root/mounts.txt
ls -l / >/mnt/root/root/lsroot.txt
ls -l /dev >/mnt/root/root/lsdev.txt
env >/mnt/root/root/env.txt
echo "All done. Switching to real root."

# clean up. The init process will remount proc sys and dev later
umount /proc
umount /sys
umount /dev

# switch to the real root and execute init
exec switch_root /mnt/root /sbin/init
FILE /usr/src/linux/initramfs/fstabvery specific fstab
/dev/sda3	/mnt/root	btrfs	device=/dev/sda3,device=/dev/sdb3,defaults,noatime,compress=lzo,autodefrag,subvol=activeroot		0 0
/dev/sda2	/mnt/boot	btrfs	device=/dev/sda2,device=/dev/sdb2,defaults,noatime		0 0
Debug boot environment files

It was interesting to see what the environment inside the kernel looks like.

FILE /root/mounts.txtResult of "cat /proc/mounts"
rootfs / rootfs rw 0 0
none /proc proc rw,relatime 0 0
none /sys sysfs rw,relatime 0 0
none /dev devtmpfs rw,relatime,size=12328636k,nr_inodes=3082159,mode=755 0 0
/dev/sda3 /mnt/root btrfs rw,noatime,compress=lzo,space_cache,autodefrag 0 0
FILE /root/lsroot.txtResult of "ls -l /" shows our initramfs as root
total 24
drwxr-xr-x    2 0        0                0 Feb 20 22:21 bin
drwxr-xr-x    8 0        0             3480 Mar  5 21:48 dev
-rwxr-xr-x    1 0        0              191 Feb 20 20:36 doit
drwxr-xr-x    2 0        0                0 Mar  5 21:48 etc
-rwxr-xr-x    1 0        0             1286 Feb 20 22:21 init
drwxr-xr-x    2 0        0                0 Feb 20 22:21 lib64
-rwxr-xr-x    1 0        0            14480 Jan 23 23:17 libdl.so.2
drwxr-xr-x    4 0        0                0 Feb 20 22:21 mnt
dr-xr-xr-x  168 0        0                0 Mar  5 21:48 proc
drwx------    2 0        0                0 Feb 20 22:21 root
drwxr-xr-x    2 0        0                0 Feb 20 22:21 sbin
dr-xr-xr-x   12 0        0                0 Mar  5 21:48 sys
drwxr-xr-x    2 0        0                0 Feb 20 22:21 usr
drwxr-xr-x    2 0        0                0 Feb 20 22:21 var
FILE /root/lsdev.txtwhat devtmpfs gives us for device tree"
total 0
crw-------    1 0        0          14,   4 Mar  5 21:48 audio
crw-------    1 0        0          10, 235 Mar  5 21:48 autofs
drwxr-xr-x    2 0        0              160 Mar  5 21:48 bsg
crw-------    1 0        0          10, 234 Mar  5 21:48 btrfs-control
drwxr-xr-x    3 0        0               60 Mar  5 21:48 bus
crw-------    1 0        0           5,   1 Mar  5 21:48 console
drwxr-xr-x   14 0        0              300 Mar  5 21:48 cpu
crw-------    1 0        0          10,  57 Mar  5 21:48 cpu_dma_latency
crw-------    1 0        0          10,  62 Mar  5 21:48 dlm-control
crw-------    1 0        0          10,  61 Mar  5 21:48 dlm-monitor
crw-------    1 0        0          10,  60 Mar  5 21:48 dlm_plock
crw-------    1 0        0          14,   3 Mar  5 21:48 dsp
crw-rw-rw-    1 0        0           1,   7 Mar  5 21:48 full
crw-------    1 0        0         247,   0 Mar  5 21:48 hidraw0
crw-------    1 0        0         247,   1 Mar  5 21:48 hidraw1
crw-------    1 0        0          10, 228 Mar  5 21:48 hpet
drwxr-xr-x    2 0        0              280 Mar  5 21:48 input
crw-------    1 0        0           1,   2 Mar  5 21:48 kmem
crw-r--r--    1 0        0           1,  11 Mar  5 21:48 kmsg
crw-------    1 0        0          10, 237 Mar  5 21:48 loop-control
brw-------    1 0        0           7,   0 Mar  5 21:48 loop0
brw-------    1 0        0           7,   1 Mar  5 21:48 loop1
brw-------    1 0        0           7,   2 Mar  5 21:48 loop2
brw-------    1 0        0           7,   3 Mar  5 21:48 loop3
brw-------    1 0        0           7,   4 Mar  5 21:48 loop4
brw-------    1 0        0           7,   5 Mar  5 21:48 loop5
brw-------    1 0        0           7,   6 Mar  5 21:48 loop6
brw-------    1 0        0           7,   7 Mar  5 21:48 loop7
drwxr-xr-x    2 0        0               60 Mar  5 21:48 mapper
crw-------    1 0        0          10, 227 Mar  5 21:48 mcelog
crw-------    1 0        0          10,  58 Mar  5 21:48 megadev0
crw-------    1 0        0           1,   1 Mar  5 21:48 mem
crw-------    1 0        0          14,   0 Mar  5 21:48 mixer
crw-------    1 0        0          10, 221 Mar  5 21:48 mpt2ctl
crw-------    1 0        0          10, 222 Mar  5 21:48 mpt3ctl
crw-------    1 0        0          10, 220 Mar  5 21:48 mptctl
crw-------    1 0        0          10,  56 Mar  5 21:48 network_latency
crw-------    1 0        0          10,  55 Mar  5 21:48 network_throughput
crw-rw-rw-    1 0        0           1,   3 Mar  5 21:48 null
crw-------    1 0        0          10, 144 Mar  5 21:48 nvram
crw-------    1 0        0           1,  12 Mar  5 21:48 oldmem
crw-------    1 0        0           1,   4 Mar  5 21:48 port
crw-rw-rw-    1 0        0           5,   2 Mar  5 21:48 ptmx
brw-------    1 0        0           1,   0 Mar  5 21:48 ram0
brw-------    1 0        0           1,   1 Mar  5 21:48 ram1
brw-------    1 0        0           1,  10 Mar  5 21:48 ram10
brw-------    1 0        0           1,  11 Mar  5 21:48 ram11
brw-------    1 0        0           1,  12 Mar  5 21:48 ram12
brw-------    1 0        0           1,  13 Mar  5 21:48 ram13
brw-------    1 0        0           1,  14 Mar  5 21:48 ram14
brw-------    1 0        0           1,  15 Mar  5 21:48 ram15
brw-------    1 0        0           1,   2 Mar  5 21:48 ram2
brw-------    1 0        0           1,   3 Mar  5 21:48 ram3
brw-------    1 0        0           1,   4 Mar  5 21:48 ram4
brw-------    1 0        0           1,   5 Mar  5 21:48 ram5
brw-------    1 0        0           1,   6 Mar  5 21:48 ram6
brw-------    1 0        0           1,   7 Mar  5 21:48 ram7
brw-------    1 0        0           1,   8 Mar  5 21:48 ram8
brw-------    1 0        0           1,   9 Mar  5 21:48 ram9
crw-rw-rw-    1 0        0           1,   8 Mar  5 21:48 random
crw-------    1 0        0         254,   0 Mar  5 21:48 rtc0
brw-------    1 0        0           8,   0 Mar  5 21:48 sda
brw-------    1 0        0           8,   1 Mar  5 21:48 sda1
brw-------    1 0        0           8,   2 Mar  5 21:48 sda2
brw-------    1 0        0           8,   3 Mar  5 21:48 sda3
brw-------    1 0        0           8,  16 Mar  5 21:48 sdb
brw-------    1 0        0           8,  17 Mar  5 21:48 sdb1
brw-------    1 0        0           8,  18 Mar  5 21:48 sdb2
brw-------    1 0        0           8,  19 Mar  5 21:48 sdb3
brw-------    1 0        0           8,  32 Mar  5 21:48 sdc
brw-------    1 0        0           8,  33 Mar  5 21:48 sdc1
brw-------    1 0        0           8,  34 Mar  5 21:48 sdc2
brw-------    1 0        0           8,  35 Mar  5 21:48 sdc3
brw-------    1 0        0           8,  48 Mar  5 21:48 sdd
brw-------    1 0        0           8,  49 Mar  5 21:48 sdd1
brw-------    1 0        0           8,  50 Mar  5 21:48 sdd2
brw-------    1 0        0           8,  51 Mar  5 21:48 sdd3
crw-------    1 0        0          14,   1 Mar  5 21:48 sequencer
crw-------    1 0        0          14,   8 Mar  5 21:48 sequencer2
crw-------    1 0        0          21,   0 Mar  5 21:48 sg0
crw-------    1 0        0          21,   1 Mar  5 21:48 sg1
crw-------    1 0        0          21,   2 Mar  5 21:48 sg2
crw-------    1 0        0          21,   3 Mar  5 21:48 sg3
crw-------    1 0        0          21,   4 Mar  5 21:48 sg4
drwxr-xr-x    2 0        0              180 Mar  5 21:48 snd
brw-------    1 0        0          11,   0 Mar  5 21:48 sr0
crw-------    1 0        0          10,  59 Mar  5 21:48 tgt
crw-rw-rw-    1 0        0           5,   0 Mar  5 21:48 tty
crw-------    1 0        0           4,   0 Mar  5 21:48 tty0
crw-------    1 0        0           4,   1 Mar  5 21:48 tty1
crw-------    1 0        0           4,  10 Mar  5 21:48 tty10
crw-------    1 0        0           4,  11 Mar  5 21:48 tty11
crw-------    1 0        0           4,  12 Mar  5 21:48 tty12
crw-------    1 0        0           4,  13 Mar  5 21:48 tty13
crw-------    1 0        0           4,  14 Mar  5 21:48 tty14
crw-------    1 0        0           4,  15 Mar  5 21:48 tty15
crw-------    1 0        0           4,  16 Mar  5 21:48 tty16
crw-------    1 0        0           4,  17 Mar  5 21:48 tty17
crw-------    1 0        0           4,  18 Mar  5 21:48 tty18
crw-------    1 0        0           4,  19 Mar  5 21:48 tty19
crw-------    1 0        0           4,   2 Mar  5 21:48 tty2
crw-------    1 0        0           4,  20 Mar  5 21:48 tty20
crw-------    1 0        0           4,  21 Mar  5 21:48 tty21
crw-------    1 0        0           4,  22 Mar  5 21:48 tty22
crw-------    1 0        0           4,  23 Mar  5 21:48 tty23
crw-------    1 0        0           4,  24 Mar  5 21:48 tty24
crw-------    1 0        0           4,  25 Mar  5 21:48 tty25
crw-------    1 0        0           4,  26 Mar  5 21:48 tty26
crw-------    1 0        0           4,  27 Mar  5 21:48 tty27
crw-------    1 0        0           4,  28 Mar  5 21:48 tty28
crw-------    1 0        0           4,  29 Mar  5 21:48 tty29
crw-------    1 0        0           4,   3 Mar  5 21:48 tty3
crw-------    1 0        0           4,  30 Mar  5 21:48 tty30
crw-------    1 0        0           4,  31 Mar  5 21:48 tty31
crw-------    1 0        0           4,  32 Mar  5 21:48 tty32
crw-------    1 0        0           4,  33 Mar  5 21:48 tty33
crw-------    1 0        0           4,  34 Mar  5 21:48 tty34
crw-------    1 0        0           4,  35 Mar  5 21:48 tty35
crw-------    1 0        0           4,  36 Mar  5 21:48 tty36
crw-------    1 0        0           4,  37 Mar  5 21:48 tty37
crw-------    1 0        0           4,  38 Mar  5 21:48 tty38
crw-------    1 0        0           4,  39 Mar  5 21:48 tty39
crw-------    1 0        0           4,   4 Mar  5 21:48 tty4
crw-------    1 0        0           4,  40 Mar  5 21:48 tty40
crw-------    1 0        0           4,  41 Mar  5 21:48 tty41
crw-------    1 0        0           4,  42 Mar  5 21:48 tty42
crw-------    1 0        0           4,  43 Mar  5 21:48 tty43
crw-------    1 0        0           4,  44 Mar  5 21:48 tty44
crw-------    1 0        0           4,  45 Mar  5 21:48 tty45
crw-------    1 0        0           4,  46 Mar  5 21:48 tty46
crw-------    1 0        0           4,  47 Mar  5 21:48 tty47
crw-------    1 0        0           4,  48 Mar  5 21:48 tty48
crw-------    1 0        0           4,  49 Mar  5 21:48 tty49
crw-------    1 0        0           4,   5 Mar  5 21:48 tty5
crw-------    1 0        0           4,  50 Mar  5 21:48 tty50
crw-------    1 0        0           4,  51 Mar  5 21:48 tty51
crw-------    1 0        0           4,  52 Mar  5 21:48 tty52
crw-------    1 0        0           4,  53 Mar  5 21:48 tty53
crw-------    1 0        0           4,  54 Mar  5 21:48 tty54
crw-------    1 0        0           4,  55 Mar  5 21:48 tty55
crw-------    1 0        0           4,  56 Mar  5 21:48 tty56
crw-------    1 0        0           4,  57 Mar  5 21:48 tty57
crw-------    1 0        0           4,  58 Mar  5 21:48 tty58
crw-------    1 0        0           4,  59 Mar  5 21:48 tty59
crw-------    1 0        0           4,   6 Mar  5 21:48 tty6
crw-------    1 0        0           4,  60 Mar  5 21:48 tty60
crw-------    1 0        0           4,  61 Mar  5 21:48 tty61
crw-------    1 0        0           4,  62 Mar  5 21:48 tty62
crw-------    1 0        0           4,  63 Mar  5 21:48 tty63
crw-------    1 0        0           4,   7 Mar  5 21:48 tty7
crw-------    1 0        0           4,   8 Mar  5 21:48 tty8
crw-------    1 0        0           4,   9 Mar  5 21:48 tty9
crw-------    1 0        0           4,  64 Mar  5 21:48 ttyS0
crw-------    1 0        0           4,  65 Mar  5 21:48 ttyS1
crw-------    1 0        0           4,  66 Mar  5 21:48 ttyS2
crw-------    1 0        0           4,  67 Mar  5 21:48 ttyS3
crw-rw-rw-    1 0        0           1,   9 Mar  5 21:48 urandom
crw-------    1 0        0         248,   0 Mar  5 21:48 usbmon0
crw-------    1 0        0         248,   1 Mar  5 21:48 usbmon1
crw-------    1 0        0         248,   2 Mar  5 21:48 usbmon2
crw-------    1 0        0         248,   3 Mar  5 21:48 usbmon3
crw-------    1 0        0         248,   4 Mar  5 21:48 usbmon4
crw-------    1 0        0         248,   5 Mar  5 21:48 usbmon5
crw-------    1 0        0         248,   6 Mar  5 21:48 usbmon6
crw-------    1 0        0         248,   7 Mar  5 21:48 usbmon7
crw-------    1 0        0         248,   8 Mar  5 21:48 usbmon8
crw-------    1 0        0           7,   0 Mar  5 21:48 vcs
crw-------    1 0        0           7,   1 Mar  5 21:48 vcs1
crw-------    1 0        0           7, 128 Mar  5 21:48 vcsa
crw-------    1 0        0           7, 129 Mar  5 21:48 vcsa1
crw-------    1 0        0          10,  63 Mar  5 21:48 vga_arbiter
crw-rw-rw-    1 0        0           1,   5 Mar  5 21:48 zero
FILE /root/env.txtbarebones root environment
HOME=/
TERM=linux
BOOT_IMAGE=/kernel-3.10.25-gentoo
PWD=/

Rebuilding kernel

root #cd /usr/src/linux
root #make -j7 bzImage
(the scripts and cpio generation are actually done pretty early on if you've already built your kernel at least once)
make[1]: Nothing to be done for `all'.
  CHK     include/generated/uapi/linux/version.h
  HOSTCC  scripts/kallsyms
  HOSTCC  scripts/pnmtologo
   .
   .
  HOSTCC  usr/gen_init_cpio
  CHK     include/generated/compile.h
  CC      init/main.o
  AS      arch/x86/ia32/ia32entry.o
  GEN     usr/initramfs_data.cpio
   .
   .
Setup is 17100 bytes (padded to 17408 bytes).
System is 11728 kB
CRC 8262ef09
Kernel: arch/x86/boot/bzImage is ready  (#3)

Our kernel just got about 3-4mb bigger as a result.

root #make -j7 modules
root #make modules_install
root #cp .config /boot/config-3.10.25-gentoo
root #cp System.map /boot/System.map-3.10.25-gentoo
root #cd arch/x86_64/boot
root #cp bzImage /boot/kernel-3.10.25-gentoo

Running grub2_mkconfig

The mkconfig will have a problem probing the multi-device set as follows:

root #grub2-mkconfig -o /boot/grub/grub.cfg
Generating grub.cfg ...
/usr/sbin/grub2-probe: error: cannot find a GRUB drive for /dev/sdc3
/dev/sdd3.  Check your device.map.
Found linux image: /boot/kernel-3.10.25-gentoo
/usr/sbin/grub2-probe: error: cannot find a GRUB drive for /dev/sdc3
/dev/sdd3.  Check your device.map.
done

This will also cause grub2-install to error out unless we uncomment and specify the GRUB_DEVICE=/dev/sdc3 in /etc/default/grub. When we googled around for this, we found a number of pages such as Fedora bug reports that suggested a patch with bash arrays and jiggered quotes in a GRUB2 script. However it doesn't appear that this patch has gotten far enough upstream or resolved to everyone's liking for Gentoo stable to pick it up yet.

We also uncomment and force GRUB_TERM=console since other pages suggest that there may be issues with GRUB2 using a framebuffer and then running the proprietary nvidia-driver for X11 later on. This may also be true for recent xf86-video-ati driver versions where we are activating a framebuffer before the GPU firmware gets loaded later on from /lib/firmware.

FILE /etc/default/grubEdited /mnt/newroot/etc/default/grub config file
# Copyright 1999-2013 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: /var/cvsroot/gentoo-x86/sys-boot/grub/files/grub.default-2,v 1.4 2013/09/21 18:10:55 floppym Exp $
#
# To populate all changes in this file you need to regenerate your
# grub configuration file afterwards:
#     'grub2-mkconfig -o /boot/grub/grub.cfg'
#
# See the grub info page for documentation on possible variables and
# their associated values. 

GRUB_DISTRIBUTOR="Gentoo"

GRUB_DEFAULT=0
GRUB_HIDDEN_TIMEOUT=0# Copyright 1999-2013 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: /var/cvsroot/gentoo-x86/sys-boot/grub/files/grub.default-2,v 1.4 2013/09/21 18:10:55 floppym Exp $
#
# To populate all changes in this file you need to regenerate your
# grub configuration file afterwards:
#     'grub2-mkconfig -o /boot/grub/grub.cfg'
#
# See the grub info page for documentation on possible variables and
# their associated values. 

GRUB_DISTRIBUTOR="Gentoo"

GRUB_DEFAULT=0
GRUB_HIDDEN_TIMEOUT=0
GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=10

# GRUB_DEVICE forces the initial root device, bypassing the autodetect
GRUB_DEVICE=/dev/sdc3

# Append parameters to the linux kernel command line
#GRUB_CMDLINE_LINUX="rootfstype=btrfs rootflags=device=/dev/sda2,device=/dev/sdb2,subvol=btroot"

# Append parameters to the linux kernel command line for non-recovery entries
#GRUB_CMDLINE_LINUX_DEFAULT=""

# Uncomment to disable graphical terminal (grub-pc only)
GRUB_TERMINAL=console

# The resolution used on graphical terminal.
# Note that you can use only modes which your graphic card supports via VBE.
# You can see them in real GRUB with the command `vbeinfo'.
#GRUB_GFXMODE=640x480

# Path to theme spec txt file.
# The starfield is by default provided with use truetype.
# NOTE: when enabling custom theme, ensure you have required font/etc.
#GRUB_THEME="/boot/grub/themes/starfield/theme.txt"

# Background image used on graphical terminal.
# Can be in various bitmap formats.
#GRUB_BACKGROUND="/boot/grub/mybackground.png"

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to kernel
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY=true

GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=10

# GRUB_DEVICE forces the initial root device, bypassing the autodetect
GRUB_DEVICE=/dev/sdc3

# Append parameters to the linux kernel command line
#GRUB_CMDLINE_LINUX="rootfstype=btrfs rootflags=device=/dev/sda2,device=/dev/sdb2,subvol=btroot"


# Append parameters to the linux kernel command line for non-recovery entries
#GRUB_CMDLINE_LINUX_DEFAULT=""

# Uncomment to disable graphical terminal (grub-pc only)
GRUB_TERMINAL=console

# The resolution used on graphical terminal.
# Note that you can use only modes which your graphic card supports via VBE.
# You can see them in real GRUB with the command `vbeinfo'.
#GRUB_GFXMODE=640x480

# Path to theme spec txt file.
# The starfield is by default provided with use truetype.
# NOTE: when enabling custom theme, ensure you have required font/etc.
#GRUB_THEME="/boot/grub/themes/starfield/theme.txt"

# Background image used on graphical terminal.
# Can be in various bitmap formats.
#GRUB_BACKGROUND="/boot/grub/mybackground.png"

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to kernel
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY=true

After the edit the mkconfig now runs without any issues.

root #grub2-mkconfig -o /boot/grub/grub.cfg
Generating grub.cfg ...
Found linux image: /boot/kernel-3.10.25-gentoo
done


The resulting grub.cfg file will specify the filesystems by uuid since we let that setting default in /etc/default/grub. We can do a sanity check as follows and compare the results.

root #btrfs fi show
Label: BTROOT  uuid: 6b9d480d-fd9f-4514-98ad-84d0ace72c90
        Total devices 2 FS bytes used 125.48GiB
        devid    1 size 1.82TiB used 128.03GiB path /dev/sdc3
        devid    2 size 1.82TiB used 128.01GiB path /dev/sdd3

Label: BTBOOT  uuid: 3dd38cf0-7591-4d26-9334-84e9f2dafddf
        Total devices 2 FS bytes used 27.41MiB
        devid    1 size 500.00MiB used 308.00MiB path /dev/sdc2
        devid    2 size 500.00MiB used 296.00MiB path /dev/sdd2

Btrfs v3.12
FILE /boot/grub/grub.cfggenerated /mnt/newroot/boot/grub/grub.cfg file
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub2-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
if [ -s $prefix/grubenv ]; then
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="0"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}

function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

terminal_input console
terminal_output console
if sleep --interruptible 0 ; then
  set timeout=10
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/10_linux ###
menuentry 'Gentoo GNU/Linux' --class gentoo --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-6b9d480d-fd9f-4514-98ad-84d0ace72c90' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_gpt
        insmod part_gpt
        insmod btrfs
        set root='hd2,gpt2'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root  3dd38cf0-7591-4d26-9334-84e9f2dafddf
        else
          search --no-floppy --fs-uuid --set=root 3dd38cf0-7591-4d26-9334-84e9f2dafddf
        fi
        echo    'Loading Linux 3.10.25-gentoo ...'
        linux   /kernel-3.10.25-gentoo root=UUID=6b9d480d-fd9f-4514-98ad-84d0ace72c90 ro rootflags=subvol=activeroot  
}
submenu 'Advanced options for Gentoo GNU/Linux' $menuentry_id_option 'gnulinux-advanced-6b9d480d-fd9f-4514-98ad-84d0ace72c90' {
        menuentry 'Gentoo GNU/Linux, with Linux 3.10.25-gentoo' --class gentoo --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-3.10.25-gentoo-advanced-6b9d480d-fd9f-4514-98ad-84d0ace72c90' {
                load_video
                set gfxpayload=keep
                insmod gzio
                insmod part_gpt
                insmod part_gpt
                insmod btrfs
                set root='hd2,gpt2'
                if [ x$feature_platform_search_hint = xy ]; then
                  search --no-floppy --fs-uuid --set=root  3dd38cf0-7591-4d26-9334-84e9f2dafddf
                else
                  search --no-floppy --fs-uuid --set=root 3dd38cf0-7591-4d26-9334-84e9f2dafddf
                fi
                echo    'Loading Linux 3.10.25-gentoo ...'
                linux   /kernel-3.10.25-gentoo root=UUID=6b9d480d-fd9f-4514-98ad-84d0ace72c90 ro rootflags=subvol=activeroot  
        }
        menuentry 'Gentoo GNU/Linux, with Linux 3.10.25-gentoo (recovery mode)' --class gentoo --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-3.10.25-gentoo-recovery-6b9d480d-fd9f-4514-98ad-84d0ace72c90' {
                load_video
                set gfxpayload=keep
                insmod gzio
                insmod part_gpt
                insmod part_gpt
                insmod btrfs
                set root='hd2,gpt2'
                if [ x$feature_platform_search_hint = xy ]; then
                  search --no-floppy --fs-uuid --set=root  3dd38cf0-7591-4d26-9334-84e9f2dafddf
                else
                  search --no-floppy --fs-uuid --set=root 3dd38cf0-7591-4d26-9334-84e9f2dafddf
                fi
                echo    'Loading Linux 3.10.25-gentoo ...'
                linux   /kernel-3.10.25-gentoo root=UUID=6b9d480d-fd9f-4514-98ad-84d0ace72c90 ro single rootflags=subvol=activeroot 
        }
}

### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_linux_xen ###
### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg;
fi
### END /etc/grub.d/41_custom ###

The set root='hd2,gpt2' statements in the grub.cfg file will be wrong once we remove the original drive set and make the new set /dev/sda and /dev/sdb. However the if-then-else statements right after do a search for the correct UUID for the BTBOOT filesystem and thus will correct the root.

Installing the MBR

We need to run grub2-install on both drives.

root #grub2-install /dev/sdc
Installation finished. No error reported.
root #grub2-install /dev/sdd
Installation finished. No error reported.

With the MBR's installed, we also want to go back and re-edit the /etc/default/grub file to reset the GRUB_DEVICE=/dev/sda3 before we forget. It may be a while before we run grub2-mkconfig again to update for a new kernel.