Project:Infrastructure/Incident reports/2021-06-28 GLIBC-2.33

Summary
New packages were rolled out that depended on glibc-2.33 symbols but we did not have glibc-2.33 merged on hosts. This resulted in programs failing to load.


 * gitweb.gentoo.org was broken for the duration of the incident.
 * grep was broken for a while.
 * gpg was broken, which prevented developers from rotating signing material.
 * git.gentoo.org was broken for a few hours and users reported failures syncing or pushing to git repositories.
 * CI was broken as the machines running CI had broken packages installed.

Fix

 * 1) Infra paused automated rollouts
 * 2) Rolled back grep (to 3.4)
 * 3) Rolled back binutils to a working version (to fix pkg_pretend for glibc binpkg install.)
 * 4) Rolled out glibc-2.33 binpkgs to the rest of the fleet

Grep
Commit f58917d806a46a40b712e184734ad1426959a249 CommitDate: Mon Jun 28 05:26:24 2021 +0000 Bumped grep from 3.4 to 3.5

sys-apps/grep3-5 requires glibc-2.33, which we were not on yet (still on 2.32.)

objdump -x /bin/grep | grep GLIBC_2.33 0x069691b3 0x00 07 GLIBC_2.33 0000000000000000      F *UND*  0000000000000000              lstat@@GLIBC_2.33 0000000000000000      F *UND*  0000000000000000              stat@@GLIBC_2.33 0000000000000000      F *UND*  0000000000000000              fstat@@GLIBC_2.33 0000000000000000      F *UND*  0000000000000000              fstatat@@GLIBC_2.33

This caused machines that merged grep-3.5 to have a broken grep. Grep is pretty important, and so this is where the breakage began.

Other packages that we observed that were busted
Other observed problems:
 * 1) binutils-2.35.2 was rolled out and also depended on glibc-2.33 symbols.
 * 2) gpg needed glibc-2.33 and was also broken on many hosts which hindered GPG key propogation.
 * 3) php on wiki was broken some of the time.
 * 4) website replication (requires gitweb.gentoo.org) was down.

Binutils ships 'as' which is used in some programs and can inhibit future package updates on the host until its resolved.