ZFS/Advanced

From Gentoo Wiki
< ZFS
Jump to:navigation Jump to:search

Overview

This section delves into advanced topics for ZFS users, encompassing a wide range of sophisticated functionalities and optimization techniques. Users will discover in-depth coverage of automation scripts that streamline routine tasks and system maintenance, along with comprehensive ZFS tuning strategies for maximizing performance across different workload scenarios. The topics include advanced dataset management, custom scripting for monitoring and maintenance, performance optimization through ARC and ZIL tuning, sophisticated backup and restoration procedures, and enterprise-level high availability configurations. Additionally, this section explores advanced encryption methodologies, RAID-Z expansion techniques, special VDEV optimization, and container integration strategies. Whether you're managing a home server or enterprise storage infrastructure, these advanced topics will help you leverage ZFS's full potential through proper system integration, automated maintenance, and performance monitoring. Particular attention is given to real-world scenarios and practical implementations, ensuring that users can effectively apply these advanced concepts to their specific use cases.

Automation Scripts

Abstraction: Manually updating the ZFS kernel module can be a time-consuming and error-prone process that involves:

  • Monitoring the OpenZFS GitHub releases page for updates
  • Checking kernel compatibility for each new release
  • Downloading and installing the appropriate versions
  • Ensuring system consistency between the ZFS module and kernel versions

System Requirements: Make sure you have the following prerequisites:

  • Linux system with bash shell
  • curl installed for downloading release information
  • Internet connection to access GitHub
  • Sufficient permissions to install kernel modules and updates
  • Directory with appropriate write permissions for script operation

Using eix post sync

The scripts below will be triggered by one script called system_update.sh. It should be executed each time the user updates their system. If there are no new updates based on the provided arguments, the script will exit. This is to ensure that the script checks for any new kernel versions, preferably after a post-sync. Create your hook script in postsync.d

root #sudo nano -w /etc/portage/postsync.d/zfs-check
FILE /etc/portage/postsync.d/zfs-checkSetting Up eix post sync
#!/bin/bash
/etc/hooks.d/zfs/system_update.sh stable 0 "/home/masterkuckles"
Note
replace stable with either testing or unknown, change 0 to 1 if you want to debug the data structures, and change "/home/masterkuckles" to where you want to copy the kernel config to

Now make it executable:

root #chmod +x /etc/portage/postsync.d/zfs-check

System Update script

Note
The programming language bash is turing complete. However, it lacks essential keywords such as try, catch, except, throw, etc. So it makes it harder to debug. Furthermore, bash will still run even if your syntax is slightly wrong, so keep that in mind. For regex parsing, visit here the functions such as error and the data structures are defined in another script
Note
There are probably some hidden edge cases that I am not aware of. I do know if the DOM tree structure gets changed, this script can break. One way to fix this would be prepending another if statement inside of parse_linux_compatibility, but you have to inspect this to find the right expression. So far: Linux: compatible with ... kernels and Linux kernels ... will get zfs-3.0 all the way down to zfs-2.2.5. If this is too confusing and you believe something is not right, you can create a pull request here after enabling DEBUG to one and running the debug_releases(), or file a bug report.
Warning
As of 2/12/25, the above script is still being tested. Most of it has been tested on a gentoo machine and works. It will install the new kernel versions based on whatever is passed in the cli ie stable, testing, unknown. There are still some bugs that need to be adressed such as installing the new zfs module, and removing the old kernel directories. Furthermore, this script has been tested only with the argument stable. It should work if you use testing or unknown. If not, you'll now what to do if the script crashes.

The script will install the latest available kernel version from the portage tree and remove any outdated kernels. If an error occurs during kernel installation, compilation, or while installing the new ZFS module, the script will catch it and exit. The script’s functionality depends on the arguments provided.

FILE /etc/hooks.d/zfs/system_update.shSystem Update
#!/bin/bash
source ./compatiability.sh 
TYPE=$(uname -m)
KEY=""
ZFS_KEY=""
STATUS="$1"
CONFIG_PATH="$3"

# Function that will check to see if there is an update or not
check_kernel() {
    # Assuming that user merged kernel source from portage
    local pkg=$(cat /var/lib/portage/world | grep -E '^sys-kernel\/[[:alpha:]]+-[[:alpha:]]+:[0-9]+\.[0-9]+\.[0-9]+$')
    if echo $pkg | grep -Eq 'gentoo-sources'; then
        # Can't assign an array to the map. Therefore, make it a string
        # and convert it back into an array using mapfile
        INSTALLED_KERNELS["gentoo-sources"]=$pkg
        KEY="gentoo-sources"
        # TODO Need to find the most latest stable version and increment it by one
        touch /etc/portage/package.mask/gentoo-sources
        #echo >=
    else
        if echo $pkg | grep -Eq 'gentoo-kernel-bin'; then
            INSTALLED_KERNELS["gentoo-kernel-bin"]=$pkg
            KEY="gentoo-kernel-bin"
            # TODO Need to find the most latest stable version and increment it by one
            touch /etc/portage/package.mask/gentoo-kernel-bin
            #echo >=
        else
            system_update_bug_report
            exit 0
        fi 
    fi
    return 0
}
# Function that will check and see if there is an update for zfs
check_zfs() {
    # Re-Update the installed kernel map
    check_kernel
    curl -s https://packages.gentoo.org/packages/sys-fs/zfs > gentoo_zfs_sources.html
    local line=$(grep -oP 'title="\d+\.\d+\.\d+ [^"]+"' gentoo_zfs_sources.html)
    local installed_zfs=$(cat /etc/portage/package.mask/zfs | sed 's/^[^a-zA-Z]*//;s/[[:space:]]*$//')
    installed_zfs=$(echo "$installed_zfs" | sed 's/[[:space:]]*$//')
    # Create an array that holds zfs and zfs-kmod package versions
    mapfile -t CURRENT_ZFS <<< "$(echo "$installed_zfs" | tr ' ' '\n')"
    local current=${CURRENT_ZFS[0]}
    while read -d '"' -r chunk; do
        if [[ $chunk =~ ^title= ]]; then
            continue  # Skip the "title=" part
        fi
        if [[ $chunk =~ ^([0-9]+\.[0-9]+\.[0-9]+)[[:space:]]is[[:space:]](testing|stable|unknown)[[:space:]]on[[:space:]]([a-z0-9]+)$ ]]; then
            version="${BASH_REMATCH[1]}"
            status="${BASH_REMATCH[2]}"
            architecture="${BASH_REMATCH[3]}"
            # Initialize if empty
            ZFS_DICT[$architecture]="${ZFS_DICT[$architecture]:-}"
            # Append new value
            ZFS_DICT[$architecture]+="$status:$version "
        fi
    done < <(echo "$line")
    local zfs_releases=""
    if [ "$TYPE" == "x86_64" ]; then
        zfs_releases=${ZFS_DICT["amd64"]}
    else 
        zfs_releases=${ZFS_DICT[$TYPE]}
    fi
    IFS=' ' read -r -a lines <<< "$zfs_releases"
    for line in "${lines[@]}"; do
        IFS=':' read -r status_val version_val <<< "$line"
        if [[ "$DEBUG" -eq 1 ]]; then
            echo "Printing out Versions:"
            echo $status_val
            echo "====================================="
 
        fi
        # Note: Issue could occur here if the DOM tree structure changes the versions around
        if [ "$STATUS" == "$status_val" ]; then 
            if [ ! -n "$ZFS_KEY" ]; then
                ZFS_KEY="zfs-$version_val"
            fi
            if [ ! -n "$current" ]; then
                current="sys-fs/zfs-$version_val"
            fi
            AVAILABLE_ZFS["$status_val"]+="sys-fs/zfs-$version_val "
        fi
    done
    # Check to see if the user has the latest version
    local new_zfs=${AVAILABLE_ZFS["$STATUS"]}
    local usr_version=$(echo "$current" | sed 's/.*-\(.*\)/\1/')
    IFS=' ' read -r -a zfs <<< "$new_zfs"
    for ele in "${zfs[@]}"; do
        available_version=$(echo "$ele" | sed 's/^[^:]*://')
        if [[ "$(echo -e "$usr_version\n$available_version" | sort -V | head -n 1)" == "$usr_version" && "$usr_version" != "$available_version" ]]; then
            echo "A zfs update is available! Going to update"
            if [ -d "/etc/portage/package.mask" ]; then 
                rm /etc/portage/package.mask/zfs
                echo ">=sys-fs/$ZFS_KEY" >> /etc/portage/package.mask/zfs
                echo ">=sys-fs/zfs-kmod-$available_version" >> /etc/portage/package.mask/zfs
            else
                echo ">=sys-fs/$ZFS_KEY" >> /etc/portage/package.mask
                echo ">=sys-fs/zfs-kmod-$available_version" >> /etc/portage/package.mask
            fi
            return 0
        else 
            echo "System has the latest version already installed!"
            exit 0
        fi
    done
    return 0
}

# Function that will install the new kernel 
install_kernel() {
    local usr_kernel_str=${INSTALLED_KERNELS[$KEY]}
    local output=0
    local copy_config=0
    for build in "${BUILD_KERNELS[@]}"; do
        version=$(echo "$build" | sed 's/^[^-]*-//')
        eselect kernel set $build
        cd /usr/src/linux || error "install_kernel" "Folder /usr/src/linux does not exist!"
        # Note: config file needs to be copied somewhere else other than home
        if [ "$copy_config" -ne 1 ]; then
            config=$(ls $CONFIG_PATH/*-config | head -n 1)
            cp -Prv $config ./.config
            copy_config=1
        fi
        if [ ! -d "/lib/modules/$version-$TYPE" ]; then 
            make menuconfig
            make -j3 || error "install_kernel" "Failed to compile" && \
            make modules_install || error "install_kernel" "Failed to install modules" && \
            make install || error "install_kernel" "Failed to install kernel"
            cp -Prv arch/x86/boot/bzImage /boot/"vmlinuz-$version-$TYPE"
        fi
        if [ -d "/usr/src/initramfs" ]; then
            new_path="$version-$TYPE"
            if [[ "$output" -eq 0 ]]; then
                echo "====================================="
                echo "Creating the directories in /usr/src/initramfs!"
                output=1
            fi
            if [ -d "/usr/src/initramfs/lib" ] && [ -d "/usr/src/initramfs/lib/modules" ]; then
                if [ ! -d "/usr/src/initramfs/lib/modules/$new_path" ]; then  
                    mkdir -vp /usr/src/initramfs/lib/modules/$new_path
                fi
            else
                if [ ! -d  "/usr/src/initramfs/lib" ]; then
                    mkdir -vp /usr/src/initramfs/lib 
                fi
                if [ ! -d "/usr/src/initramfs/lib/modules" ]; then 
                    mkdir -vp /usr/src/initramfs/lib/modules
                fi
                if [ ! -d "/usr/src/initramfs/lib/modules/$new_path" ]; then 
                    mkdir -vp /usr/src/initramfs/lib/modules/$new_path
                fi
            fi
            if [ ! -d "/usr/src/initramfs/lib/modules/$new_path/extra" ]; then
                echo "Creating the extra folder to copy over the modules!"
                mkdir -vp /usr/src/initramfs/lib/modules/$new_path/extra
            fi
        fi
    done
    mapfile -t usr_kernel_arr <<< "$usr_kernel_str"
    local clean_up=0
    for usr_kernels in "${usr_kernel_arr[@]}"; do
        if [ "$clean_up" -ne 0 ]; then
            echo "====================================="
            echo "Cleaning up old directories in /usr/src/initramfs/lib/modules!"
            echo "====================================="
            clean_up=1
        fi
        # Strip off the package name and colon to extract just the version
        usr_version=$(echo "$usr_kernels" | sed 's/^[^:]*://')
        if [ -d "/usr/src/initramfs/$usr_version-gentoo-$TYPE" ]; then
            if [ -d "/usr/src/initramfs/lib/modules/$usr_version-gentoo-$TYPE" ]; then 
                echo "====================================="
                echo "Removing old folders from /usr/src/initramfs/modules/$usr_version-gentoo-$TYPE"
                rm -r /usr/src/initramfs/lib/modules/"$usr_version-gentoo-$TYPE" || error "install_kernel" "Failed to remove $usr_version-gentoo-$TYPE from /usr/src/initramfs/lib/modules/"
            fi
            if [ -d "/lib/modules/$usr_version-gentoo-$TYPE" ]; then 
                echo "====================================="
                echo "Removing old folders from /lib/modules/$usr_version-gentoo-$TYPE"
                rm -r /lib/modules/"$usr_version-gentoo-$TYPE" || error "install_kernel" "Failed to remove $usr_version-gentoo-$TYPE from /lib/modules/"
            fi 
            if [ -d "/boot/vmlinuz-$usr_version-gentoo-$TYPE" ]; then 
                echo "====================================="
                echo "Removing the old vmlinuz from /boot/vmlinuz-$usr_version-gentoo-$TYPE"
                rm  /boot/"vmlinuz-$usr_version-gentoo-$TYPE" || error "install_kernel" "Failed to remove vmlinuz-$usr_version-$TYPE from /boot"
            fi 
            if [ -d "/boot/initramfs-$usr_version-$TYPE.img" ]; then 
                echo "====================================="
                echo "Removing the old initramfs from /boot/initramfs-$usr_version-$TYPE.img"
                rm /boot/"initramfs-$usr_version-$TYPE.img" || error "install_kernel" "Failed to remove initramfs-$usr_version-$TYPE.img from /boot"
            fi 
        else 
            if [ -d "/lib/modules/$usr_version-gentoo-$TYPE" ]; then 
                echo "====================================="
                echo "Removing old folders from /lib/modules/$usr_version-gentoo-$TYPE"
                rm -r /lib/modules/"$usr_version-gentoo-$TYPE" || error "install_kernel" "Failed to remove $usr_version-gentoo-$TYPE from /lib/modules/"
            fi 
            if [ -d "/boot/vmlinuz-$usr_version-gentoo-$TYPE" ]; then 
                echo "====================================="
                echo "Removing the old vmlinuz from /boot/vmlinuz-$usr_version-gentoo-$TYPE"
                rm  /boot/"vmlinuz-$usr_version-gentoo-$TYPE" || error "install_kernel" "Failed to remove vmlinuz-$usr_version-$TYPE from /boot"
            fi 
            if [ -d "/boot/initramfs-$usr_version-$TYPE.img" ]; then 
                echo "====================================="
                echo "Removing the old initramfs from /boot/initramfs-$usr_version-$TYPE.img"
                rm /boot/"initramfs-$usr_version-$TYPE.img" || error "install_kernel" "Failed to remove initramfs-$usr_version-$TYPE.img from /boot"
            fi 
        fi
    done
    return 0
}

# Function will only install the specific version based on the argument that is passed in before run time
install_kernel_resources() {
    local usr_kernel_str=${INSTALLED_KERNELS[$KEY]}
    local available_kernel_str=""
    local build_str=""
    if [ "$TYPE" == "x86_64" ]; then
        available_kernel_str=${ARCH_DICT["amd64"]}
    else 
        available_kernel_str=${ARCH_DICT[$TYPE]}
    fi
    if [ ! -n "$available_kernel_str" ] || [ ! -n "$usr_kernel_str" ]; then
        system_update_bug_report
        exit 0
    fi
    # Split the lines at white-spaces
    IFS=' ' read -r -a lines <<< "$available_kernel_str"
    for line in "${lines[@]}"; do
        IFS=':' read -r status_val version_val <<< "$line"
        if [[ "$DEBUG" -eq 1 ]]; then
            echo "Printing out Versions:"
            echo $status_val
            echo "====================================="
        fi
        if [ "$STATUS" == "$status_val" ]; then 
            AVAILABLE_KERNELS["$status_val"]+="sys-kernel/$KEY:$version_val "
        fi
    done
    mapfile -t usr_kernel_arr <<< "$usr_kernel_str"
    IFS=' ' read -r -a kernel_arr <<< "${AVAILABLE_KERNELS["$STATUS"]}"
    local copy_config=0
    local kernel_update=0
    for usr_kernels in "${usr_kernel_arr[@]}"; do
        # Strip off the package name and colon to extract just the version
        usr_version=$(echo "$usr_kernels" | sed 's/^[^:]*://')
        if [ ! -n "$usr_version" ]; then
            system_update_bug_report
            exit 0
        fi
        for available_kernels in "${kernel_arr[@]}"; do
            # Strip off the package name and colon to extract just the version
            available_version=$(echo "$available_kernels" | sed 's/^[^:]*://')
            if [ ! -n "$available_kernels" ]; then
                system_update_bug_report
                exit 0
            fi
            if [[ "$(echo -e "$usr_version\n$available_version" | sort -V | head -n 1)" == "$usr_version" && "$usr_version" != "$available_version" ]]; then
                if [[ ! "$build_str" =~ "linux-$available_version-gentoo" ]]; then
                    build_str+="linux-$available_version-gentoo "
                fi
                # Note: config file needs to be copied somewhere else other than home
                if [ "$copy_config" -ne 1 ]; then
                    copy_config=1
                    kernel_update=1
                    cp -Prv /usr/src/"linux-$usr_version-gentoo"/.config $CONFIG_PATH/"$usr_version-gentoo-$TYPE-config"
                fi
                if ! equery list =sys-kernel/"$KEY-$available_version" > /dev/null; then 
                    emerge -v =sys-kernel/"$KEY-$available_version" || error "install_kernel_resources" "Failed to install the new kernel!"
                    emerge --deselect =sys-kernel/"$KEY-$available_version" || error "install_kernel_resources" "Failed to deselect the old kernels"
                    if [ -d "/usr/src/linux-$available_version-gentoo" ]; then 
                        rm -r  /usr/src/"linux-$available_version-gentoo"
                    fi
		        else 
			        echo "Kernel $KEY-available_version is already installed"
		        fi
            fi
        done 
    done
    if [ "$kernel_update" -ne 0 ]; then
        # Remove trailing spaces and newlines
        build_str=$(echo "$build_str" | sed 's/[[:space:]]*$//')
        mapfile -d ' ' -t BUILD_KERNELS <<< "$build_str"
        install_kernel
        emerge -a --depclean
    else 
        echo "====================================="
        echo "Already have the latest kernel version installed!"
        echo "Exiting script..."
        echo "====================================="
        clean_up
        exit 0
    fi
    return 0
}

# Function that will update to the newest zfs
install_zfs() {
    local compatible_kernel_ranges=${COMPATIBLE_RELEASES[$ZFS_KEY]}
    local installed_kernels_str=${INSTALLED_KERNELS[$KEY]}
    local start=$(echo "$compatible_kernel_ranges" | cut -d '-' -f1)
    local end=$(echo "$compatible_kernel_ranges" | sed 's/^[^-]*-//')
    mapfile -t kernels <<< "$installed_kernels_str"
    for ele in "${kernels[@]}"; do
        # Strip off the package name and colon to extract just the version
        version=$(echo "$ele" | sed 's/^[^:]*://')
        if version_greater_than_equal "$version" "$start" && version_less_than_equal "$version" "$end"; then
            new_path="$version-$TYPE"
            zfs_version=$(echo $ZFS_KEY | sed 's/^[^-]*-//')
            echo "Kernel version $kernel is within the range ($start - $end)."
            emerge --deselect ${CURRENT_ZFS[0]} || error "install_zfs" "Failed to uninstall the old zfs"
            emerge --deselect ${CURRENT_ZFS[1]} || error "install_zfs" "Failed to uninstall the old zfs kmod"
            emerge -1 =sys-fs/$ZFS_KEY || error "install_zfs" "Failed to install the newer zfs!"
            emerge -1 =sys-fs/zfs"-"kmod"-"$zfs_version || error "install_zfs" "Failed to install the new zfs-kmod!"
            if [ -d "/usr/src/initramfs" ]; then 
                cp -Prv /lib/modules/$new_path/extra/* /usr/src/initramfs/lib/modules/$new_path/extra/ || error "install_zfs" "Failed to copy over modules!"
                cp -Prv /lib/modules/$new_path/modules.* /usr/src/initramfs/lib/modules/$new_path/
            fi 
        else
            echo "====================================="
            echo "Kernel version $kernel is outside the range ($start - $end)."
        fi
    done 
    return 0
}
update_init() {
    local usr_kernel_str=${INSTALLED_KERNELS[$KEY]}
    mapfile -t usr_kernel_arr <<< "$usr_kernel_str"
    if [ -d "/usr/src/initramfs" ]; then 
        cd /usr/src/initramfs
    fi 
    for usr_kernels in "${usr_kernel_arr[@]}"; do
        usr_version=$(echo "$usr_kernels" | sed 's/^[^:]*://')
        if [ -d "/usr/src/initramfs/lib/modules/$usr_version-gentoo-$TYPE" ]; then
            lddtree --copy-to-tree /usr/src/initramfs /sbin/zfs 
            lddtree --copy-to-tree /usr/src/initramfs /sbin/zpool 
            lddtree --copy-to-tree /usr/src/initramfs /sbin/zed 
            lddtree --copy-to-tree /usr/src/initramfs /sbin/zgenhostid
            lddtree --copy-to-tree /usr/src/initramfs /sbin/zvol_wait 
            find . -not -path "/lib/modules/*" -o -path "./lib/modules/$usr_version-gentoo-$TYPE/*" -print0 | cpio --null --create --verbose --format=newc | gzip -9 > boot/initramfs-"$usr_version-gentoo-$TYPE".img
            echo "====================================="
        else 
            Dracut=$(cat /var/lib/portage/world | grep -E '^sys-kernel\/dracut-+:[0-9]+\.[0-9]+\.[0-9]+$')
            # Need to check to see if the string is empty
            if [[ -n "$Dracut" ]]; then
                # If it is not empty then generate initramfs using dracut
                dracut --force --kver="$usr_version" /boot/initramfs-"$usr_version-gentoo-$TYPE".img || error "update_init" "Failed to create initramfs using dracut!"
            fi
            GenKernel=$(cat /var/lib/portage/world | grep -E '^sys-kernel\/genkernel-+:[0-9]+\.[0-9]+\.[0-9]+$')
            # Need to check to see if the string is empty
            if [[ -n "$GenKernel" ]]; then
                # If it is not empty then generate initramfs using genkernel
                genkernel --kernel-config=/usr/src/linux-$usr_version-gentoo/.config initramfs --kerneldir=/usr/src/linux-$usr_version-gentoo || error "update_init" "Failed to create initramfs using genkernel!"
            else 
                error "update_init" "Error: The 'Dracut' and 'GenKernel' variables are empty.\nThis indicates a bug in the script. Please file a bug report or submit a pull request to resolve the issue at: https://github.com/alphastigma101/BashScripts\n\nSteps to follow:\n1. Provide details of your environment (e.g., OS, Bash version).\n2. Describe how to reproduce the issue.\n3. Submit a pull request with a fix if possible.\n\nIf you are unable to submit a fix, please report the issue with as much detail as you can."
            fi  
        fi 
    done 
    return 0
}

populate_data_structures() {
    curl -s https://packages.gentoo.org/packages/sys-kernel/$KEY > kernel_sources.html
    local line=$(grep -oP 'title="\d+\.\d+\.\d+ [^"]+"' kernel_sources.html)
    while read -d '"' -r chunk; do
        if [[ $chunk =~ ^title= ]]; then
            continue  # Skip the "title=" part
        fi
        if [[ $chunk =~ ^([0-9]+\.[0-9]+\.[0-9]+)[[:space:]]is[[:space:]](testing|stable|unknown)[[:space:]]on[[:space:]]([a-z0-9]+)$ ]]; then
            version="${BASH_REMATCH[1]}"
            status="${BASH_REMATCH[2]}"
            architecture="${BASH_REMATCH[3]}"
            # Initialize if empty
            ARCH_DICT[$architecture]="${ARCH_DICT[$architecture]:-}"
            # Append new value
            ARCH_DICT[$architecture]+="$status:$version "
        fi
    done < <(echo "$line")
    return 0
}

clean_up() {
    remove_html 
    return 0
}

main() {
    populate_zfs_releases
    check_kernel
    populate_data_structures
    install_kernel_resources "$STATUS"
    check_zfs
    install_zfs
    update_init
    update_bootloader
    clean_up
}
main
if [[ "$DEBUG" -eq 1 ]]; then
    debug_releases
    debug_arch_dict
    debug_installed_kernel
    debug_stable_kernel
    debug_testing_kernel
fi

compatibility script

This script web-scrapes this page to find the most recent updated ZFS module and the supported kernel version that goes with it. Keep in mind that the latest zfs module that is supported on that github openzfs will always have a stable version higher than the gentoo's current one. It would be very wise to use the zfs module that gentoo has marked down as stable.

Note
Its functions and data structures are used in another script, which will be explained further down this section.
CODE Compataibility
#!/bin/bash
source ./bug_report.sh
source ./debug_data_structures.sh 
source ./helper_functions.sh
source ./error_handling.sh
DEBUG="$2"

# Associative array to store compatible releases
declare -A COMPATIBLE_RELEASES
declare -a RANGES
# Function to extract Linux kernel compatibility
parse_linux_compatibility() {
    local url="$1"
    local release_name="$2"
    curl -s $url > $release_name".html"
    # Use grep and sed to extract Linux kernel compatibility line
    local linux_line=$(grep -A10 "Supported Platforms:" "$release_name.html" | grep "Linux kernels" | sed -e 's/^\s*//; s/\s*$//')
    # Check if Linux compatibility line exists
    if [[ -z "$linux_line" ]]; then
        linux_line=$(grep -A10 "Supported Platforms" "$release_name.html" | grep -oP "Linux: compatible with [0-9.]+ - [0-9.]+ kernels" | sed -e 's/^\s*//; s/\s*$//')
    fi
    # Extract the smallest and largest kernel versions using a regex
    if [[ "$linux_line" =~ Linux:\ compatible\ with\ ([0-9.]+)\ -\ ([0-9.]+)\ kernels ]]; then
        min_version="${BASH_REMATCH[1]}"
        max_version="${BASH_REMATCH[2]}"
    fi
    if [[ "$linux_line" =~ Linux\ kernels\ ([0-9]+\.[0-9]+)\ -\ ([0-9]+\.[0-9]+) ]]; then
        min_version="${BASH_REMATCH[1]}"
        max_version="${BASH_REMATCH[2]}"
    fi
    # Store the result in RANGES and COMPATIBLE_RELEASES
    RANGES[0]="$min_version"
    RANGES[1]="$max_version"
    COMPATIBLE_RELEASES["$release_name"]="${RANGES[0]}-${RANGES[1]}"
    return 1
}

populate_releases() {
    # Fetch the releases page content
    curl -s https://github.com/openzfs/zfs/releases > zfs_releases.html
    mapfile -t RELEASES_PAGE < <(
        awk 'BEGIN{RS="<div"; ORS="<div"} /class="Box-body"/{flag=1} flag; /<\/div>/{flag=0}' zfs_releases.html | \
        grep -oP 'href="\/openzfs\/zfs\/releases\/tag\/[^"]+"' | cut -d '"' -f 2
    )
    for url in "${RELEASES_PAGE[@]}"; do
        release_name=$(echo $url | grep -oP 'zfs-.*$')
        # Try to extract Linux kernel compatibility
        parse_linux_compatibility "https://github.com$url" "$release_name"
    done
}
debug_releases() {
    # Print out the compatible releases
    echo "Compatible ZFS Releases:"
    for release in "${!COMPATIBLE_RELEASES[@]}"; do
        echo "$release: Linux kernel ${COMPATIBLE_RELEASES[$release]}"
    done
}
remove_html() {
    rm -f *.html
}
# Check if DEBUG is set to 1
if [[ "$DEBUG" -eq 1 ]]; then
    # Call populate_releases function if DEBUG is enabled
    populate_releases
    debug_releases
fi

Bug Report script

If something crashes during run time, this script will be executed by calling a certain function. It will provide in depth details as what you can do. Keep in mind that this script can become very brittle and break if any of the DOM structures of the web-pages that are being web-scraped change.

FILE /etc/hooks.d/zfs/bug_report.shBug Report
#!/bin/bash
system_update_bug_report() {
    echo -e "### Bug Report: Issue with Kernel Update Detection in bash script\n"
    echo -e "#### Description:\n"
    echo -e "The script fails to correctly parse or match kernel packages ending with numeric versions. This issue affects the detection of installed kernel versions when running the script for Gentoo systems. Specifically, `gentoo-sources` or other kernel packages with numeric versions are not being detected properly, which impacts further operations in the script.\n"
    echo -e "#### Steps to Reproduce:\n"
    echo -e "#### How to Report a Bug:\n"
    echo -e "1. Visit the repository issues page: https://github.com/alphastigma101/BashScripts/issues\n"
    echo -e "2. Click on 'New Issue' to create a bug report.\n"
    echo -e "3. Provide a detailed description of the issue, including steps to reproduce the error and the expected vs. actual behavior.\n"
    echo -e "4. Include the relevant logs or output from running the script, as well as any error messages.\n"
    echo -e "5. Be sure to include your system and environment details such as Gentoo version, Portage version, and Bash version.\n"
    echo -e "\n#### Expected Behavior:\n"
    echo -e "The script should correctly parse and list installed kernel versions with numeric versioning, specifically matching lines like 'sys-kernel/gentoo-sources:6.1.118'.\n"
    echo -e "\n#### Actual Behavior:\n"
    echo -e "The script fails to detect or properly handle kernel packages that end with numeric versions (e.g., 'sys-kernel/gentoo-sources:6.1.118') and does not store them in the dictionary.\n"
    echo -e "\n#### Environment:\n"
    echo -e " - Operating System: Gentoo Linux\n"
    echo -e " - Script Version: $(git describe --tags)\n"
    echo -e " - Bash version: $(bash --version | head -n 1)\n"
    echo -e " - Portage version: $(emerge --info | grep -i 'portage' | head -n 1)\n"
    echo -e "\n#### Logs/Output:\n"
    echo -e "Please provide any additional log output below that could help identify the root cause of the issue.\n"
    echo -e "For example:\n"
    echo -e " - Output from running the kernel update script.\n"
    echo -e " - Any error messages.\n"
}

Debug Data Structures script

This script provides a list of implemented functions that can debug the data structures if DEBUG is set to 1. This is useful when unexcpected behavior occurs. And one of the reasons could be that there's no data inside a specific data structure, which will cause the script to break.

FILE /etc/hooks.d/zfs/debug_data_structures.shDebug Data Structures
#!/bin/bash
declare -A INSTALLED_KERNELS
declare -A ARCH_DICT
declare -A ZFS_DICT
declare -A STABLE_KERNELS
declare -A TESTING_KERNELS
declare -A AVAILABLE_KERNELS
declare -A AVAILABLE_ZFS
declare -a BUILD_KERNELS
declare -a CURRENT_ZFS

debug_zfs_releases() {
    # Print out the compatible releases
    echo "Compatible ZFS Releases:"
    for release in "${!COMPATIBLE_RELEASES[@]}"; do
        echo "$release: Linux kernel ${COMPATIBLE_RELEASES[$release]}"
    done
    echo "====================================="
}

debug_arch_dict() {
    for arch in "${!ARCH_DICT[@]}"; do
        echo "Architecture: $arch"
        echo "Versions: ${ARCH_DICT[$arch]}"
        echo "-------------------"  
    done
    echo "====================================="
    return 0
}

debug_installed_kernel() {
    echo "Kernel Packages:"
    for key in "${!INSTALLED_KERNELS[@]}"; do
        echo "$key: ${INSTALLED_KERNELS[$key]}"
    done
    echo "====================================="
    return 0
}

debug_available_kernel() {
    echo "AVAILABLE_KERNELS:"
    for key in "${!AVAILABLE_KERNELS[@]}"; do
        echo "$key: ${AVAILABLE_KERNELS[$key]}"
    done
    return 0
}

Helper Functions script

Contains a series of functions that are used inside system_update.sh or else where. Tha main reason why I created this script is to make the code more organize and easier to maintain.

FILE /etc/portage/postsync.d/helper_functions.shHelper Functions
#!/bin/bash

# Function to compare if version is greater than or equal to the comparison version
version_greater_than_equal() {
    [[ "$(echo -e "$1\n$2" | sort -V | head -n 1)" == "$2" ]]
}

# Function to compare if version is less than or equal to the comparison version
version_less_than_equal() {
    [[ "$(echo -e "$1\n$2" | sort -V | tail -n 1)" == "$2" ]]
}

Error Handling script

If something happens during run time, this the string literal of the function name where unexpected behaivor occured and the a brief sentence explaining what happen. Keep in mind, some errors can be false due to missing folders or files. Which isn't a problem, it just means that another if statement is needed.

FILE /etc/portage/postsync.d/error_handling.shError Handling
#!/bin/bash

# Function that will output errors
error() {
    local x="$1"
    local y="$2"
    local arr="$3"
    if [ "$x" == "install_kernel" ]; then
        echo "[ERROR:] In $x. $y! "
        echo "Cleaning up...."
        make clean || cd /usr/src/linux && make clean
        make mrproper
        exit 0
    elif  [ "$x" == "install_kernel_resources" ]; then
        echo "[ERROR:] In $x. $y! "
        exit 0
    elif [ "$x" == "install_zfs" ]; then 
        echo "[ERROR:] In $x. $y! "
        exit 0
    elif [ "$x" == "update_init" ]; then 
        echo "[ERROR:] In $x. $y! "
        exit 0
    fi
    exit 0
}

Finialization

After everything is implemented and configured, make the scripts inside /etc/hooks.d/zfs executable:

root #chmod +x /etc/hooks.d/zfs/*.sh