Project:Infrastructure/Incident reports/2021-06-28 GLIBC-2.33
New packages were rolled out that depended on glibc-2.33 symbols but we did not have glibc-2.33 merged on hosts. This resulted in programs failing to load.
- gitweb.gentoo.org was broken for the duration of the incident.
- grep was broken for a while.
- gpg was broken, which prevented developers from rotating signing material.
- git.gentoo.org was broken for a few hours and users reported failures syncing or pushing to git repositories.
- CI was broken as the machines running CI had broken packages installed.
- Infra paused automated rollouts
- Rolled back grep (to 3.4)
- Rolled back binutils to a working version (to fix pkg_pretend for glibc binpkg install.)
- Rolled out glibc-2.33 binpkgs to the rest of the fleet
Commit f58917d806a46a40b712e184734ad1426959a249 CommitDate: Mon Jun 28 05:26:24 2021 +0000 Bumped grep from 3.4 to 3.5
sys-apps/grep3-5 requires glibc-2.33, which we were not on yet (still on 2.32.)
objdump -x /bin/grep | grep GLIBC_2.33 0x069691b3 0x00 07 GLIBC_2.33 0000000000000000 F *UND* 0000000000000000 lstat@@GLIBC_2.33 0000000000000000 F *UND* 0000000000000000 stat@@GLIBC_2.33 0000000000000000 F *UND* 0000000000000000 fstat@@GLIBC_2.33 0000000000000000 F *UND* 0000000000000000 fstatat@@GLIBC_2.33
This caused machines that merged grep-3.5 to have a broken grep. Grep is pretty important, and so this is where the breakage began.
Other packages that we observed that were busted
Other observed problems:
- binutils-2.35.2 was rolled out and also depended on glibc-2.33 symbols.
- gpg needed glibc-2.33 and was also broken on many hosts which hindered GPG key propogation.
- php on wiki was broken some of the time.
- website replication (requires gitweb.gentoo.org) was down.
Binutils ships 'as' which is used in some programs and can inhibit future package updates on the host until its resolved.