Project:RelEng GRS

Description
The Gentoo Reference System (GRS) Suite is a set of tools for building and maintaining a well defined Gentoo system in which all choices in building the system are predefined in configuration files housed on a central git repository. All systems built according to a particular GRS spec should be identical. As a "from source" distribution, Gentoo allows a large degree of customization. The space of all possible packages and USE flags is vast, not to speak of more radical choices such as different kernels (eg. Linux or FreeBSD), executable formats (eg. ELF or Mach-O), different C Standard Libraries (eg. glibc, uClibc or musl) and different providers of core userland utilities (eg. busybox or coreutils). In contrast to other distributions where each instance of an installed system is nearly identical to another, the large array of choice in Gentoo means that any two systems are unlikely to be sufficiently similar that executables or libraries from one will "just work" on the other, even if the architecture and other hardware factors are the same; assuming, of course, there is no conscious effort to build identical systems. This is where the Gentoo Release System (GRS) Suite comes in. It does precisely this, namely, it provides an automated method for repeatedly and predictably generating identical Gentoo systems.

GRS is designed to work roughly as follows: Periodic release tarballs are generated which are "polished". For example, releases can provide pre-configured display managers, windowing systems and desktop themes, even user accounts and home directories. Installation should be as simple as unpacking the release tarball on pre-formated partitions with minimal or no further configuration. There is no need to build any packages or a kernel and everything is ready to go out of the box. A release tarball can be downloaded from some server or alternatively can be built locally. While these may not initially be identical because they were build at different times, after updating, both approaches should yield identical systems.

Updating a GRS system can proceed by either building packages locally, or downloading pre-built binpkgs. As long as one does not deviate from the GRS defined set of USE flags, maskings and other configuration parameters, both approaches should yield identical systems. A GRS system is always a Gentoo system, so at any time, one can elect to part ways from GRS and venture out on one's own! The GRS Suite provides a utilities to make sure that configurations in /etc/portage are properly maintained in a manner consistent with GRS, but emerge and other portage utilities will always work. Even if one does deviate from the GRS specs, it should be possible to return to strict GRS using the Suite's utilities, provided one has not deviated too far.

GRS is provided by the app-portage/grs package. The same package is installed on either a server responsible for building the release tarballs and binpkgs, or on an actual GRS system. On the server, a daemon called grsrun will either do a release run, in which case it builds the release tarballs, or it will do an update run where it either builds or updates a bunch of extra binpkgs. For example, for GRS = desktop-amd64-uclibc-hardened, the release run builds a little under 900 packages and produces the polished release tarball, while the update run builds/updates about 5700 packages. The first update run after a new release is time consuming because some 5700 new packages must be built, but subsequent update runs need only build packages which were bumped since the last update run.

On the client, a utility called grsup acts as a wrapper to emerge. grsup will both manage the configuration files in /etc/portage as well as either builds or download the binpkg from a PORTAGE_BINHOST. After the initial installation of a GRS system, one need only run grsup to bring it up to date. While releases tarballs will be pushed out periodically, these are not used to update an existing GRS system, only to start new one. Rather, one GRS release should smoothly update to the next; in other words, each GRS release is equivalent to the previous release plus any updates since. The only reason to push out a new release tarball is to avoid having to subsequently push out a large number of updates for each new installation.

CAVEAT: This is work in progress. A few of the above point are marginally working or implemented. This warning will disappear after this whole project moves past being experimental.

Features:


 * The GRS suite does not hard code any steps for the release or update runs. Rather, a build script on the git repository describes the steps for building each particular GRS system.  This makes GRS runs highly flexible.  One need only transcribe the steps one would manually make in a chroot to build the system into build script directives, and grsrun will automatically repeat them.


 * It is possible to script a GRS system to do the equivalent of catalyst runs. [TODO: there is still one more build script directives required here, pivot_chroot.]


 * The use of Linux cgroup make management easy. There is no need to trap or propagate signals when writing the scripts that are to be run in the chroot.  A simple SIGTERM to grsrun will ensure that all children process no matter how deep are properly terminated and that any bind-mounted filesystems are unmounted.


 * A GRS system acts as a "tinderbox lite" in that build failures are reported. So as a GRS system evolves over time, as package are bumped, any breakages that are introduced will be caught and reported. [TODO: get these reports automatically into bugzilla.]

Build a release with
The GRS build suite aims to be very flexible, so rather than overwhelm ourselves with possibilities, we'll go through one example scenario in detail. In this section, we'll be the maintainer of a GRS system and make use of the  command to build a fully featured desktop system on a binhost server. Our final product for release will be a tarball of the system, complete with an already compiled kernel, pre-configured windowing system and a polished default user home and desktop, that is ready to run out-of-the-box. Just untar onto pre-formatted filesystems, install the bootloader, and reboot into a pristine and well polished system. In the next section, we'll use the  command to maintain an installed instance of our desktop. Maintaining a GRS system with  is completely optional and one can just treat it as any old Gentoo system, but if you want to follow the GRS spec as they change upstream, you need to use. Its a wrapper on  which makes sure you don't deviate from the specs for your GRS system. But we're getting ahead of ourselves.

All the information to build a GRS system lives on one branch of a git repository. As of this writing there is only one such repository https://gitweb.gentoo.org/proj/grs.git/ with four branches:

desktop-amd64-musl-hardened desktop-amd64-hardened desktop-amd64-uclibc-hardened stages-amd64-hardened

Each branch is dedicated to one particular GRS system, so let's take the desktop-amd64-musl-hardened system as our example. There's nothing special about it, and the fact that it is based on musl rather than glibc is not really relevant. If we check out that branch, we find the following at the top level:

build core/ README scripts/ TODO

The  script is the most important file there since it contains the directives to   for what it is supposed to do to build the system. It is the only file at the top level directory that  will look at and you are free to put other things there, like a   or , since these will be ignored. They are there for human consumption.

The  directory contains any files that are going to be copied into the new system. They are just copied directly into the GRS root. Any file, anywhere in the filesystem, with any content, can be copied in, clobbering whatever is already there. In our case, that directory contains the following (only partially reproduced here):

core/ ├── etc │   ├── avahi │   │   └── avahi-daemon.conf │   ├── conf.d       │   │   ├── hostname │   │   └── xdm │   ├── fstab │   ├── grs │   │   ├── systems.conf │   │   └── world.conf │   ├── lilo.conf │   ├── modprobe.d       │   │   └── blacklist.conf │   ├── portage │   │   ├── make.conf.CYCLE.1 │   │   ├── make.conf.CYCLE.2 │   │   ├── make.conf.CYCLE.3 │   │   ├── make.profile -> ../../usr/portage/profiles/hardened/linux/musl/amd64 │   │   ├── package.accept_keywords │   │   │   ├── app-portage_grs_0 │   │   │   ├── dev-lib_libcgroup_0 │   │   │   └── sys-libs_fts-standalone_0 │   │   └── repos.conf │   │      └── gentoo.conf ...       │   ├── slim.conf │   ├── sudoers │   └── X11 │      └── xorg.conf ...       ├── usr │   └── share │      └── slim │          └── themes │              └── default │                  └── background.jpg ...       └── var └── lib └── portage ├── world.CYCLE.1 ├── world.CYCLE.2 ├── world.CYCLE.3 └── world.CYCLE.4

You'll notice that there are quite a few files in /etc which pre-configure the system to make sure it works out-of-the-box. Eg1. We have /etc/conf.d/xdm configured to use slim as our desktop greeter, we have /etc/slim.conf set up so slim will behave a particular way and present a paricular theme and start up XFCE4 on login, and (not shown in the above listing) we have /etc/skel and one regular user's home directory set up with a nicely configured desktop once XFCE4 is running. Eg2. musl doesn't have lazy linking, so configuring X is even more painful than usual. We include an xorg.conf which should just work.

You'll also notice that there are several copies of make.conf and world. The  allows you to build your system in stages called "cycles". When you copy your files from  to the root of your GRS system, you can select which cycle you want. If a file doesn't have a cycle number, then it is constant across cycles. But if it ends in .CYCLE.X, then it is copied in only when cycle X is selected. Of course, the .CYCLE.X suffix is removed. Here X must be an integer since there a particular behavior when you select a cycle higher than your highest X --- this will make more sense when we look at the  directive below.

The  directory contains scripts to be run in the GRS root as it is built. Think of these scripts as snippets of what you would type as you grow your stage3 into the system you want, like the steps you taken in the Handbook. These scripts should be written in either bash or python since all stage3's come with these interpreters, but you could use any scripting language like perl if you've already emerged perl at an early point in the build process.

In our example, these scripts are used in conjunction with the cycles we specify in our  directory, but there is no necessary connection between cycles and scripts. So our build process goes something like this: copy and possibly overwrite some files for a particular cycle from  to the GRS root, run one of our scripts, copy and possibly overwrite some other files for a different cycle from   and then run some other scripts. Repeat until userland is done. Let look at this in detail:

At cycle 1, we start with an empty  file and a bland   which does nothing more than repeat the CFLAGS, CHOST etc from the stage3 seed. Once these are in place, we run  in the GRS root which basically adds the musl overlay. Of course to do so, you need to emerge layman first since that doesn't ship with our stage3, so our script looks like this:

Its just a few bash commands which achieve one goal: add the musl overlay to the stage3. How we downloaded the stage3, unpacked it, etc., we'll see below. Right now, just understand what this little piece accomplishes even if you don't see the full context.

At cycle 2, we add a world file of some 65 "most dependant" packages (ie packages which depend on other packages, but which are not themselves depended upon). These packages defines our desktop's @world set. We expand our make.conf to define some more portage variables like LINGUAS, source our overlay which we added in the prevous step and we rebuild all of @world. Our  looks as follows (modulo some details):

In this script, we first rebuild the toolchain. This is probably not necessary, but since musl is experimental we do it just to be sure it works (and we have the cpu cycles to murder!). Then we rebuild all of @world. By the way, notice the -b flag to emerge because we want to store the .tbz2 files which we'll serve out from the binhost. There's a bit more to this script which we'll come back to in the next section when we discuss.

Next at cycle 3, we change the world file and update our USE flags slightly. If you've ever build a large system from a stage3, you've probably hit circular dependencies. These can be thorny from a logical point of view and so portage is not able to disentangle them ... yet. We resolve these much the same way you resolve them manually, you build with some set of USE flags that avoids the circular dependency and then add the flags you want back and rebuild. In our case, we need to avoid the old avahi-cups-zeroconf circularity (See |bug #222601), so we first build without cups and then rebuild with cups. Modulo some details we'll return to, our  only has

to do an update with the new flags. We don't need a full @world rebuild.

At this point you might think that all the scripts in the  directory are about emerging components of your system, but this is not case. You can add services to runlevels, set up users and groups and to do any other post installation configurations as would if you were building your own custom system --- except that here it is all scripted are reproduceable. The following are some extra scripts for our desktop-amd64-musl-hardened system:

and

and

With above techniques, you can build and configure pretty much any userland you want for your GRS system. However, to have a system ready to go out-of-the-box, we'll also need a pre-compiled kernel, so  is capable of providing that. builds the kernel external to the GRS root using the host's toolchain, packs it as a .tar.xz and installs it into the GRS root. The pre-compiled kernel package is stored in the  directory as a file called linux-images/linux-image- - .tar.xz so it can be installed on a remote GRS system using. (Note: means which sources, you're using, like 'gentoo', 'hardened' or 'vanilla'.) You'll probably want this if you want to provide packages for pre-compiled kernel updates.

Once userland and the kernel are built and installed, the GRS system root is finished. will create the tarball and digests to prepare it for shipping. At the user end, all that needs to be done is to extract the tarball over pre-formatted filesystems, do a quick chroot to install lilo (or whatever bootloader the GRS system comes with), and finally reboot into a pristine, well polished system. This is basically how Lilblue and the Lemote Yeelong Desktop are installed.

The process, however, doesn't have to stop here. If you want to provide additional binpkgs beyond those shipped with your release tarball, you can have  continue to use the GRS root to build them. These binpkgs are stored in the  directory on the binhost and can be installed on local GRS systems either using   or.

This completes our overview but we're still missing some important details, so let's revisit the above from the point of view of the real heart of the GRS specs, the  script at the top level of the GRS git repo. It orchestrates the entire process with a very simple set of commands called 'directives'. Since  can install any number of cycles, or run any number of scripts in any order, you might wonder how does it know when to do what. That's what the  script is for, so let's take a look at the build script for desktop-amd64-musl-hardened:

The directives here are 'log', 'mount', 'umount', 'populate', 'runscript', 'kernel', 'tarit' and 'hashit'. Let's describe each:


 * log: The 'log' directive does exactly that. It places a message and a time stamp in  's log which is found at  .  The name of the log file follows the name of the GRS system and it records all the output from the scripts run in the GRS root.  This is useful for debugging what's going on in the chroot as you build the system.  However, there is also another log generated called   which records any exceptions thrown by the daemon spawned by   to handle the build of this particular GRS system.  This log is useful for debugging errors in the GRS suite itself.  Both logs are auto-rotated on (re)starting up the daemon.


 * mount: This directive will mount all the necessary directories below the GRS root. Unlike catalyst which creates a snapshot of the portage tree before bind-mounting it under the chroot,   will mount the host's portage tree directly.  This is a lot faster but also dangerous because changes to the portage tree while a run is in progress can adversely affect the system being built.  Its recommended that   be run on a dedicated server where you can synchronize changes to the portage tree with a run.  [TODO: add an option to mount which allows for snapshotting of the portage tree.]


 * umount: This directive does the opposite of 'mount'. Since   sets up its own cgroup, it tracks all process which might have open file descriptors on mounted filesystems.  If any step fails,   will clean up and umount itself.  So you don't need to use 'umount' defensively, only when you are really done with the mounted filesystems under your chroot.  In the above example, we mount before we run scripts  ,   and   because these invoke   which requires the portage tree be present.  However we 'umount' before packaging up our system for shipping because we don't want the portage tree in our tarball.  And if, say,   fails,   will umount everything for you just before it terminates.


 * populate X: This directive copies the files from the  directory of the GRS git repo into the GRS root, overwriting any files in the chroot.  The integer X corresponds to the cycle number to selects the files as described above.  If there are a sequence of cycled files, and X is bigger than any of the cycle numbers on the file, then the file with the largest cycle number is used.  Eg. if the cycle of files is ,  ,   and X = 4 or greater, then   is copied in as.


 * runscript X: This directive will run the 'X' script in the GRS root. The above examples have the .sh suffix to indicate that they are shell scripts, but no suffix is really necessary, just a shebang on the first line.  In fact, there is no restriction on the naming of scripts whatsoever.  If you are starting with a stage3 tarball, 'X' can be in either bash or python.  If, however, you emerged, say, perl in a previous step, then 'X' can be in perl.


 * kernel: This directive looks for a kernel configuration file in the GRS git repo at  and uses it to build a kernel image and modules.  The version of the kernel is obtained from the third line of the config file, so that:

# Linux/x86 4.0.6-hardened-r2 Kernel Configuration

would use =sys-kernel/hardened-sources-4.0.6-r2 while

# Linux/x86 4.0.5-gentoo Kernel Configuration

would use =sys-kernel/gentoo-sources-4.0.5. The kernel is built using the host's toolchain, not the GRS's toolchain, and the image and modules are bundled as linux-images/linux-image-4.0.6-hardened-r2.tar.xz under the  directory as well as is installed into the GRS root. Remember, if you want to target a large variety of hardware, you'll want to turn on as many modules and features as possible in the kernel.


 * tarit [ X ]: This directive will tar everything in the GRS root found under the  will name the tarball by the GRS system name followed by a time stamp.  In our case, this would be something like desktop-amd64-musl-hardened-20150721.tar.xz.  However, if X is supplied, the GRS name is replace by X.  For example, if we have 'tarit stage3-amd64-hardened' then the tarball will be named stage3-amd64-hardened-20150721.tar.xz instead.


 * hashit: This will create an MD5, SHA1, SHA512 and WHIRLPOOL has of the tarball and place them in a file whose name is the same as the tarball name with suffix .DIGESTS. Both the tarball and digest are placed in the GRS's work directory found at /var/tmp/grs/desktop-amd64-musl-hardened for our example.

So in sum, the  script orchistrates the entire build process from the initial seed (usually, but not necessarily, a stage3 tarball) until the final release tarball. However you'll notice a few steps after the tarball is mastered and the digest produced. The purpose of these lines is to build the extra binpks beyond what is released in the tarball, as we mentioned above. So in this case 'populate 4' expands the world file to include some 5000 packages and  does an `emerge -uvNDq @world` to build the added packages. You'll also notice that these lines begin with +'s.   operates in two modes: 1) Without any flag   will just do all the steps in the   script in order.   This is 'release' mode and its used when preparing another release of the GRS system.  2) With the -u flag,   runs in 'update' mode and skips all the lines without +'s.  By marking only those lines needed to build updates, this mode is useful for generating binpkgs between releases.

We need to mention that an actual GRS run involves two 'invisible' directives before the build script directives are initiated. These are the 'sync' and 'seed' step. You don't need to, and indeed can't, specify them in your  script, as they just happen automatically. When  is invoked, it reads its configuration file at   which specifies what GRS systems you're are going to build. (TODO: add a flag which allows you to redirect to alt config file) Here's an example of the file:

Its in configparser format where each section defines a GRS system to build. The repo_uri line gives the location of the GRS git repo where a branch matching the section name is to be found. During the 'sync' step, the git repo is cloned into /var/lib/grs/desktop-amd64-musl-hardened and then the desktop-amd64-muls-hardened branch is checked out. The later build directives will look into this directory for files to populate or scripts to run. A similar process takes place for each of the GRS systems specified in the various sections of the config file.

The 'seed' step follows immediately after 'sync' and it will download the stage tarball specified on the stage_uri line. It will place it a /var/tmp/grs/desktop-amd64-musl-hardened and unpack it. If it finds the stage tarball already there, it won't download it again. (TODO: check the MD5 to make sure its not broken.)

Since multiple GRS systems can be specified in the  file, you may wonder how does   juggle multiple runs? When you run, it will build each of the specified systems in parallel, spawning a daemon for each and isolating the files and process. So desktop-amd64-musl-hardened will have

and similarly for desktop-amd64-uclibc-hardened and any other GRS systems specified. The daemons for each GRS system are (hopefully!) well behaved and you cannot spawn two daemons for the same GRS system running at the same time. That would be bad because the build process is assumed to be serial in the  script and the two daemons would step over each other.

Once  has sync-ed and seeded, it can proceed to build the GRS system according to the directives in the build script. As each step is completed a  is placed in   directory. Here XX is a two digit number referring line in the build script that completed successfully. The excepiton here is the 'sync' and 'seed' steps for which XX = 'sync' or 'seed' respectively. This show the progress but also allows you to pick up where you left off if something should go wrong. If  is restarted, it will skip any steps for which it sees a   file. If, for example, you identify the problem to be in the GRS git repo, and fix it there, you delete, rerun   and it will repeat the 'sync' step, skip everything in between, and go to where you left off. You may, of course, have to also undelete some other steps along with the 'sync', like the 'mount' step, to make sure you have the right environment to pick up where you left off properly.