GLEP:58

Abstract
MetaManifest provides a means of verifiable distribution from Gentoo Infrastructure to a user system, while data is conveyed over completely untrusted networks and system, by extending the Manifest2 specification, and adding a top-level Manifest file, with support for other nested Manifests.

Motivation
As part of a comprehensive security plan, we need a way to prove that something originating from Gentoo as an organization (read Gentoo-owned hardware, run by infrastructure), has not been tampered with. This allows the usage of third-party rsync mirrors, without worrying that they have modified something critical (e.g. eclasses, which are still unsigned).

Securing the untrusted distribution is one of the easier tasks in the security plan - in short, all that is required is having a hash of every item in the tree, and signing that hash to prove it came from Gentoo.

Ironically we have a hashed and signed distribution (it's just not used by most users, due to it's drawbacks): Our tree snapshot tarballs have hashes and signatures.

So now we want to add the same verification to our material that is distributed by rsync. We already provide hashes of subsets of the tree - our Manifests protect individual packages. However metadata, eclasses and profiles are not protected at this time. The directories of packages and distfiles are NOT covered by this, as they are not distributed by rsync.

This portion of the tree-signing work provides only the following guarantee: A user can prove that the tree from the Gentoo infrastructure has not been tampered with since leaving the Gentoo infrastructure. No other guarantees, either implicit or explicit are made.

Additionally, distributing a set of the most recent MetaManifests from a trusted source allows validation of trees that come from community mirrors, and allows detection of all cases of malicious mirrors (either by deliberate delay, replay or alteration).

Specification
For lack of a better name, the following solution should be known as the MetaManifest. Those responsible for the name have already been sacked.

MetaManifest basically contains hashes of every file in the tree, either directly or indirectly. The direct case applies to ANY file that does not appear in an existing Manifest file (e.g. eclasses, Manifest files themselves). The indirect case is covered by the CONTENTS of existing Manifest files. If the Manifest itself is correct, we know that by tracking the hash of the Manifest, we can be assured that the contents are protected.

In the following, the MetaManifest file is a file named 'Manifest', located at the root of a repository.

Summary
The objective of creating the MetaManifest file(s) is to ensure that every single file in the tree occurs in at least one Manifest.

Process

 * 1) Start at the root of the Gentoo Portage tree (gentoo-x86, although this procedure applies to overlays as well).
 * 2) Initialize two unordered sets: COVERED, ALL.
 * 3) 'ALL' shall contain every file that exists in the present tree.
 * 4) 'COVERED' shall contain EVERY file that is mentioned in an existing Manifest2. If a file is mentioned in a Manifest2, but does not exist, it must still be included. No files should be excluded.
 * 5) Traverse the tree, depth-first.
 * 6) At the top level only, ignore the following directories: distfiles, packages, local.
 * 7) If a directory contains a Manifest file, extract all relevant local files from it (presently: AUX, MISC, EBUILD; but should follow the evolution of Manifest2 entry types per ), and place them into the COVERED set.
 * 8) Recursively add every file in the directory to the ALL set, pursuant to the exclusion list as mentioned in.
 * 9) Produce a new set, UNCOVERED, as the set-difference (ALL)-(COVERED). This is every item that is not covered by another Manifest, or part of an exclusion list.
 * 10) If an existing MetaManifest file is present, remove it.
 * 11) For each file in UNCOVERED, assign a Manifest2 type, produce the hashes, and add with the filetype to the MetaManifest file.
 * 12) For unique identification of the MetaManifest, a header line should be included, using the exact contents of the metadata/timestamp.x file, so that a MetaManifest may be tied back to a tree as  distributed by the rsync mirror system. The string of 'metadata/timestamp.x' should be included to identify this revision of MetaManifest generation. e.g.: "Timestamp: metadata/timestamp.x: 1215722461 Thu Jul 10 20:41:01 2008 UTC" The package manager MUST not use the identifying string as a filename.
 * 13) The MetaManifest must ultimately be GnuPG-signed.
 * 14) For the initial implementation, the same key as used for snapshot tarball signing is sufficient.
 * 15) For the future, the key used for fully automated signing by infra should not be on the same keyring as developer keys. See  for further notes.

Verification of one or more items from the MetaManifest
There are two times that this may happen: firstly, immediately after the rsync has completed - this has the advantage that the kernel file cache is hot, and checking the entire tree can be accomplished quickly. Secondly, the MetaManifest should be checked during installation of a package.

Procedure for verifying an item in the MetaManifest
In the following, I've used term 'M2-verify' to note following the hash verification procedures as defined by the Manifest2 format - which compromise checking the file length, and that the hashes match. Which filetypes may be ignored on missing is discussed in GLEP 60.


 * 1) Check the GnuPG signature on the MetaManifest against the keyring of automated Gentoo keys. See GLEPxx3 for full details regarding verification of GnuPG signatures.
 * 2) Abort if the signature check fails.
 * 3) Check the Timestamp header. If it is significantly out of date compared to the local clock or a trusted source, halt or require manual intervention from the user.
 * 4) For a verification of the tree following an rsync:
 * 5) Build a set 'ALL' of every file covered by the rsync. (exclude distfiles/, packages/, local/)
 * 6) M2-verify every entry in the MetaManifest, descending into inferior Manifests as needed. Place the relative path of every checked item into a set 'COVERED'.
 * 7) Construct the set 'UNCOVERED' by set-difference between the ALL and COVERED sets.
 * 8) For each file in the UNCOVERED set, assign a Manifest2 filetype.
 * 9) If the filetype for any file in the UNCOVERED set requires a halt on error, abort and display a suitable error.
 * 10) Completed verification
 * 11) If checking at the installation of a package:
 * 12) M2-verify the entry in MetaManifest for the Manifest
 * 13) M2-verify all relevant metadata/ contents if metadata/ is being used in any way (optionally done before dependency checking).
 * 14) M2-verifying the contents of the Manifest.
 * 15) Perform M2-verification of all eclasses and profiles used (both directly and indirectly) by the ebuild.

Implementation Notes
For this portion of the tree-signing work, no actions are required of the individual Gentoo developers. They will continue to develop and commit as they do presently, and the MetaManifest is added by Infrastructure during the tree generation process, and distributed to users.

Any scripts generating Manifests and the MetaManifest may find it useful to generate multiple levels of Manifests in parallel, and this is explicitly permitted, provided that every file in the tree is covered by at least one Manifest or the MetaManifest file. The uppermost Manifest (MetaManifest) is the only item that does not occur in any other Manifest file, but is instead GPG-signed to enable its validation.

MetaManifest and the new Manifest2 filetypes
While GLEP 60 describes the addition of new filetypes, these are NOT needed for implementation of the MetaManifest proposal. Without the new filetypes, all entries in the MetaManifest would be of type 'MISC'.

Timestamps & Additional distribution of MetaManifest
As discussed by, malicious third-party mirrors may use the principles of exclusion and replay to deny an update to clients, while at the same time recording the identity of clients to attack.

This should be guarded against by including a timestamp in the header of the MetaManifest, as well as distributing the latest MetaManifests by a trusted channel.

On all rsync mirrors directly maintained by the Gentoo infrastructure, and not on community mirrors, there should be a new module 'gentoo-portage-metamanifests'. Within this module, all MetaManifests for a recent time frame (e.g. one week) should be kept, named as "MetaManifest.$TS", where $TS is the timestamp from inside the file. The most recent MetaManifest should always be symlinked as MetaManifest.current. The possibility of serving the recent MetaManifests via HTTPS should also be explored to mitigate man-in-the-middle attacks.

The package manager should obtain MetaManifest.current and use it to decide is the tree is too out of date per operation #2 of the verification process. The decision about freshness should be a user-configuration setting, with the ability to override.

MetaManifest size considerations
With only two levels of Manifests (per-package and top-level), every rsync will cause a lot of traffic transferring the modified top-level MetaManifest. To reduce this, first-level directory Manifests are required. Alternatively, if the distribution method efficiently handles small patch-like changes in an existing file, using an uncompressed MetaManifest may be acceptable (this would primarily be distributed version control systems). Other suggestions in reducing this traffic are welcomed.

Backwards Compatibility

 * There are no backwards compatibility issues, as old versions of Portage do not look for a Manifest file at the top level of the tree.
 * Manifest2-aware versions of Portage ignore all entries that they are not certain how to handle. Enabling headers and PGP signing to be conducted easily.

Thanks
I'd like to thank the following people for input on this GLEP.
 * Patrick Lauer (patrick): Prodding me to get all of the tree-signing work finished, and helping to edit.
 * Ciaran McCreesh (ciaranm): Paludis Manifest2
 * Brian Harring (ferringb): pkgcore Manifest2
 * Marius Mauch (genone) & Zac Medico (zmedico): Portage Manifest2
 * Ned Ludd (solar) - Security concept review

Copyright
Copyright (c) 2006-2010 by Robin Hugh Johnson. This material may be distributed only subject to the terms and conditions set forth in the Open Publication License, v1.0.