Bisecting with live ebuilds

This article has been flagged for not conforming to the wiki guidelines (has introduction section, not following blueprint). Please help Gentoo out by starting fixing things.

Suppose one hits a regression in ZFS. In this example, between ZFS 2.1.4 (last known good version) and ZFS 2.1.6 (being the latest version where a bug is hit).

Using live ebuilds and git bisect can be enormously helpful for debugging issues. This debugging tactic approach is possible for any ebuilds with a version number of *-9999.

Steps

First, clone ZFS at the zfs-2.1.5-staging branch:

user $git clone https://github.com/openzfs/zfs -b zfs-2.1.5-staging

Start the bisect:

user $git bisect start

user $git bisect bad zfs-2.1.5

user $git bisect good zfs-2.1.4

Create a simple script to allow easily building each commit:

FILE /tmp/test-zfs.sh

export EGIT_OVERRIDE_COMMIT_OPENZFS_ZFS=$(git rev-parse HEAD)
export EGIT_OVERRIDE_BRANCH_OPENZFS_ZFS=zfs-2.1.5-staging
export ACCEPT_KEYWORDS="**"
emerge -v1 zfs zfs-kmod

echo "If this version is good (need to reboot), run 'git bisect good'."
echo "If this version is bad (need to reboot), run 'git bisect bad'."

Run /tmp/test-zfs.sh repeatedly.

After each run of emerge by the script, test out ZFS by rebooting. If it worked, type git bisect good. If the bug remained, type git bisect bad. Continue until the first bad commit is identified.

Using git bisect run

git bisect has a 'run' subcommand which allows running a command or script to determine if a commit is good or bad.

See https://lwn.net/Articles/317154/.

For example, if sys-libs/glibc is failing in its last-minute sanity-check in pkg_preinst, it's sufficient to use the following script with the glibc git repository:

FILE /tmp/test-glibc.sh

#!/bin/bash
export EGIT_OVERRIDE_COMMIT_GLIBC=$(git rev-parse HEAD)
export EGIT_OVERRIDE_BRANCH_GLIBC=release/2.35/master
export ACCEPT_KEYWORDS="**"
#USE="vanilla" ...
emerge -v1 sys-libs/glibc || exit 1

GCC example

When filing GCC bugs upstream, it's common to need to do two things:

1. Bisect to when it starts failing
2. Minimize a reproducer

For bisecting, the following script may be useful:

FILE /tmp/gcc-bisect.sh

#!/usr/bin/env bash
# Use with 'git bisect run /tmp/gcc-bisect.sh' in a checkout of gcc.git
# GPL-3+
. /lib/gentoo/functions.sh || { echo "Failed to source functions.sh!" ; exit 1 ; }

export ACCEPT_KEYWORDS="**"
export FEATURES="ccache -collision-protect -protect-owned -userfetch -usersync -userpriv -usersandbox"
#export EVCS_OFFLINE=1
export EGIT_CLONE_TYPE=mirror
# Path to clone of repo
export EGIT_OVERRIDE_REPO_GCC=file:///home/sam/git/gcc
export EGIT_OVERRIDE_COMMIT_GCC=$(git rev-parse HEAD)

esyslog() {
        : No need.
}

die() {
        eerror "$@"
        exit 255 # Abort the bisect process
}

fatal() {
        einfo Running "$@"
        "$@" || die "$@"
}

skip() {
        einfo Running "$@"
        if ! "$@"; then
                ewarn Build failed, skipping: "$@"
                exit 125
        fi
}

gcc_ver=$(<gcc/BASE-VER)
gcc_ver=${gcc_ver%%.*}

fatal cd ~/git/gcc

# If it might flip-flop between major versions, need
# the following to pick a working version first
# because of the symlink shenanigans below:
#fatal gcc-config powerpc-unknown-linux-gnu-10
#fatal source /etc/profile

# Extra checking may be useful if debugging a difficult issue.
# --enable-checking=all
EXTRA_ECONF="--disable-bootstrap" USE="vanilla -rust -openmp -fortran -pgo -lto -jit" skip ebuild /var/db/repos/gentoo/sys-devel/gcc/gcc-${gcc_ver}.*.9999.ebuild digest clean merge

# Try hard to make sure we're actually using the newly-built thing
# (only needed if the commits flip-flop between major versions)
#gcc-config powerpc-unknown-linux-gnu-${gcc_ver}
#source /etc/profile
#export PATH="/root/bisect/bld/_pfx/bin:$PATH"
#export CC=gcc-${gcc_ver}
#export CXX=g++-${gcc_ver}

# Build Binutils with each GCC commit and consider the commit bad if Binutils fails to compile.
# Replace this with whatever the relevant test condition is for a commit being bad/good.
USE="-pgo" fatal ebuild /var/db/repos/gentoo/sys-devel/binutils/binutils-2.40.ebuild clean compile install

exit 0

See also

• Kernel git-bisect