Project:Infrastructure/Incident reports/2021-06-28 GLIBC-2.33

Summary

New packages were rolled out that depended on glibc-2.33 symbols but we did not have glibc-2.33 merged on hosts. This resulted in programs failing to load.

• gitweb.gentoo.org was broken for the duration of the incident.
• grep was broken for a while.
• gpg was broken, which prevented developers from rotating signing material.
• git.gentoo.org was broken for a few hours and users reported failures syncing or pushing to git repositories.
• CI was broken as the machines running CI had broken packages installed.

Fix

1. Infra paused automated rollouts
2. Rolled back grep (to 3.4)
3. Rolled back binutils to a working version (to fix pkg_pretend for glibc binpkg install.)
4. Rolled out glibc-2.33 binpkgs to the rest of the fleet

Grep

Commit f58917d806a46a40b712e184734ad1426959a249
CommitDate: Mon Jun 28 05:26:24 2021 +0000
Bumped grep from 3.4 to 3.5

sys-apps/grep3-5 requires glibc-2.33, which we were not on yet (still on 2.32.)

objdump -x /bin/grep | grep GLIBC_2.33
  0x069691b3 0x00 07 GLIBC_2.33
0000000000000000       F *UND*  0000000000000000              lstat@@GLIBC_2.33
0000000000000000       F *UND*  0000000000000000              stat@@GLIBC_2.33
0000000000000000       F *UND*  0000000000000000              fstat@@GLIBC_2.33
0000000000000000       F *UND*  0000000000000000              fstatat@@GLIBC_2.33

This caused machines that merged grep-3.5 to have a broken grep. Grep is pretty important, and so this is where the breakage began.

Other packages that we observed that were busted

Other observed problems:

1. binutils-2.35.2 was rolled out and also depended on glibc-2.33 symbols.
2. gpg needed glibc-2.33 and was also broken on many hosts which hindered GPG key propogation.
3. php on wiki was broken some of the time.
4. website replication (requires gitweb.gentoo.org) was down.

Binutils ships 'as' which is used in some programs and can inhibit future package updates on the host until its resolved.