User:Sakaki/Sakaki's EFI Install Guide/Building the Gentoo Base System Minus Kernel

In this section, we'll be following along with Chapter 6 of the Gentoo handbook, and installing the base system. However, we'll also be diverging considerably, as we are going to be using systemd (not Gentoo's more usual OpenRC) as the init system (since GNOME 3.8+ requires this ), and bootstrapping the core system files (apart from the kernel, which will be addressed in the next section).

The process we'll be following in this section is:
 * 1) Performing final preparations for a chroot (setting mirrors, copying DNS, and mounting required system directories);
 * 2) Entering the chroot environment (after which, the  location will appear to be the true root directory of the filesystem);
 * 3) Installing an up-to-date Portage tree;
 * 4) Ensuring that the base profile is set;
 * 5) Setting your timezone and locale;
 * 6) Installing a tool to show parallel builds in progress, from the sakaki-tools overlay;
 * 7) Bootstrapping the toolchain (optional);
 * 8) Rebuilding everything in the @world set using the new toolchain (optional);
 * 9) Verifying that all executables and libraries have been rebuilt (optional);
 * 10) Setting the final target profile (for GNOME/systemd), and updating the the @world set to reflect this.

Instructions will also be provided for those who don't want to spend the time necessary to perform a full bootstrap. Let's get started!

Final Preparations for the chroot
We begin by setting up appropriate the server URIs, which portage will search when looking for source tarballs. To do this, we use the mirrorselect tool (shipped as part of the minimal install system), which allows you to choose the appropriate mirror from a list, and then save the result to the file (as an assignment to the GENTOO_MIRRORS</tt> variable):

Next, setup a file (this specifies information about repositories in the system to Portage; see the earlier overview for a brief primer on the concepts involved). Issue:

and edit the file so it reads:

Save and exit nano</tt>.

The above simply says that:
 * The main repository is set to be gentoo</tt>, for all other repositories (such as overlays) that do not specify masters;
 * The repository location is set to be (within the chroot</tt>, that is);
 * The repository is set to synchronize using the rsync</tt> protocol.

Next, we must specify the sync-uri</tt> variable, which tells Portage where to look for the rsync</tt> server, when bringing your Portage tree of ebuilds up to date.

Issue:

Check that something appropriate was written to and :

Next up, we need to <span id="copy_dns_info">make sure our DNS information, which exists on our target machine's current host system at, is still available after we chroot</tt>; otherwise networking will stop working. Issue:

The -L</tt> option ensures we don't copy a symbolic link by mistake, since the host filesystem will become inaccessible once we chroot</tt>.

Next, <span id="copy_wpa_conf">if you are performing this install over a WiFi network, copy the configuration file for wpa_supplicant</tt> (which we setup earlier) into the chroot</tt> filesystem. Issue:

Finally, we need to <span id="setup_bind_mounts">ensure that the kernel information in, and the special files in and  are still available from inside the chroot</tt>. We therefore mount on, and then bind-mount  and :

<span id="enter_chroot">Entering the chroot</tt>
We're now ready to change root (chroot</tt>), thereby entering our new system. After this is done, although we will still be running the minimal install kernel pro tem, commands issued from within the chroot</tt>-ed shell will only be able to 'see' and use files (including, but not limited to, executables, libraries, and configuration files) that lie within the stage 3 file structure we just unpacked (except for the special, and  directories, of course).

Per the handbook, we'll perform the chroot</tt> in three steps:
 * 1) Enter the <tt>chroot</tt>, using bash as the shell;
 * 2) Reload some settings, using <tt>source</tt>; and
 * 3) Redefine the console prompt (to remind us we are in a <tt>chroot</tt>).

Issue:

<span id="chroot_prompt">then :

<span id="update_portage_tree">Installing an Up-to-Date Portage Tree
The next step is to install a Portage (repository tree) snapshot, a set of files that tells Portage what software is available to install, what profiles are available, and so on. These updates are produced on a daily basis. Issue:

The fetching may take some time to complete. Once done, the <tt>emerge-webrsync</tt> command may recommend some software be updated, and tell you there is news to read: if so, you can safely ignore these for now (we're about to do a ground-up rebuild anyway, and we'll get to the news items shortly).

Next, we'll bring the Portage tree bang up to date (this may also take a little time to complete):

Next, make sure that Portage itself is up-to-date:

The <tt>emerge</tt> command, without special options, asks Portage to pull down (if necessary) and install the latest version of the package specified, in this case the Portage system itself. The <tt>--oneshot</tt> option informs Portage not to add itself to the set of apps in the 'world' file for later updating. The <tt>--ask --verbose</tt> flag set makes <tt>emerge</tt> ask you before making any changes to your system, and produce verbose output (generally useful, and may be abbreviated to <tt>-av</tt>).

Now, if that <tt>emerge</tt> worked OK, (you eventually see "Jobs: 1 of 1 complete" in the output after you pressed and ), congratulations: your Portage system is functional, and you should be able to proceed.

Finally, <span id="reading_news">if you were prompted about news items being available, please take the time to read them now: these are important, short announcements from developers to users, pushed out via the Portage tree. To view the current news items issue:

You'll notice the items are numbered. To read a news item N, enter:

You can purge all read news items with:

If you like, you can read more about the Gentoo news system on its manual page (using <tt>man news.eselect</tt>).

<span id="check_base_profile">Ensuring that the Base Profile is Set
As was mentioned earlier, Gentoo uses profiles to set important variables (such as <tt>USE</tt>, <tt>CFLAGS</tt>, etc.) relevant to your particular architecture and usage scenario (e.g., a machine running the GNOME desktop). Profiles also constrain the versions of packages that can be installed. This is all maintained on your behalf by the Gentoo developers.

At this point, we just need to check that we are running the baseline <tt>amd64</tt> profile (we should be). We'll change this to the 'real' profile post-bootstrap. Issue:

And double-check that the first entry is marked with an asterisk, indicating that it is active. If not, issue:

to correct this.

You can see the various (cumulative) variables assignments (to <tt>USE</tt> etc.) associated with a profile by looking at its file, and those of its parents. Specifically, the <tt>eselect profile set N</tt> command creates a symbolic link in. If you follow this link, and then look at the files in that directory (and in any of the directories listed by the  text file therein (transitively)), you will see how it works. Note that <tt>USE</tt> flags accumulate, but a flag with a minus sign in front of it gets removed from the list.

You can also see the net effect of the profile, your, environment etc. by issuing (this is a useful sanity check):

Lastly, if you are unsure of the meaning of any use flag, you can look it up as follows:

<span id="set_timezone">Setting Up Timezone and Locale
We don't have a timezone or locale set yet. So let's fix that now.

First up is timezone. You can find the available timezone in (generally, you'll have to check the subdirectory too):

Choose an appropriate location, and place its relative path into. For example, to set <span id="set_europe_london">London, in the UK, you would use:

Now, we need to reconfigure the package; this will pick up the value from  and reflect the correct value in. The latter is necessary for the system C library.

Secondly, we must <span id="modify_locale_gen">set an appropriate locale. You must specify the locales you want to use in. Edit this file (using your favourite editor, such as <tt>nano</tt>), and uncomment those that you want. If necessary, add additional locales to the end of the list. For example, as there are no UK locales on the list (UK = United Kingdom, approximately the same ^-^ as Great Britain = GB) use we would need to add those as follows:

Then append (for this example):

Save and exit the <tt>nano</tt> editor.

Next, we must run <tt>locale-gen</tt> to create the locales, based on the information in the file:

Assuming that completes successfully, you then must then specify the default locale to use. Find the system list with:

Choose the appropriate value from the list. At this stage, I recommend you stick with the 'C' (default) locale (other setups can cause issues with the indirect <tt>ssh</tt>/<tt>screen</tt> remote connection); we'll switch to your actual locale later when we cut over to <tt>systemd</tt> later on. Issue:

Now, reload your environment:

<span id="prep_for_parallel_emerge">Preparing to Run Parallel <tt>emerge</tt>s
We have (intentionally) set our Portage system up to maximize parallelism when emerging. However, a side-effect of this is that, because multi-job support is active, <tt>emerge</tt> does not show any console build output, just a summary progress display.

This is easily fixed, using a simple script, which will show us the latest lines from both the Portage background tarball download log and from the most recently updated build log (Portage builds its projects in  by default). I have provided this and some other necessary utilities for this tutorial in an overlay, which makes them straightforward to install and keep up to date within Portage (for details of overlays, see the background reading section earlier).

To begin with, we'll first install the <tt>layman</tt> tool, as this greatly simplifies overlay management. Issue:

Next, we need to ensure that Portage will check the overlay locations managed by <tt>layman</tt> (in effect setting the <tt>PORTDIR_OVERLAY</tt> variable). Issue:

Next, we'll download and display a list of the overlays that <tt>layman</tt> currently knows about. Issue:

However, as the overlay we need (<tt>sakaki-tools</tt>) is not in this list, we need to add it. Information about additional remote overlays may be specified via xml files stored in, so issue:

Then add the following text into the (new) file (the contents should be self-explanatory):

Save and exit the <tt>nano</tt> editor.

Now we must synchronize and display the overlay list again, and verify that <tt>sakaki-tools</tt> is now visible (this is important to do, or the subsequent <tt>layman --add</tt> command may fail). Issue once more:

Now, we need to ask <tt>layman</tt> to activate the overlay, which will download its <tt>ebuild</tt> repository (no software will be installed from the new overlay yet, however). Issue:

Whenever using a third-party overlay such as this one, remember that it will (in general) take precedence over your 'real' Portage tree (the one it 'overlays'). For this reason, it is prudent in such cases to:
 * 1) mask (blacklist) all packages from the overlay, then
 * 2) unmask (whitelist) only those packages from the overlay you explicitly know you want, then
 * 3) read the contents of those remaining, unmasked overlay ebuilds before installing (emerging) them.

We'll begin by wildcard-masking everything from <tt>sakaki-tools</tt>. As described earlier, we use the file for this. Issue:

Now we can unmask what we explicitly need (using, as described earlier, the file). Issue:

Then append the following text into the file:

Save and exit the <tt>nano</tt> editor.

Here is a brief summary of what the above (whitelisted) ebuilds are, and what they do:

Next, note that all user overlay ebuilds (by stipulation) must specify that they are on the 'unstable' branch (since not yet fully tested). However, we are allowing (by default) only stable (<tt>amd64</tt>) packages (see above), we have to specify that for the above packages, the unstable/testing ('tilde') branch is acceptable. We use the file for this, as described earlier. Issue:

Then append the following text into the file:

Save and exit the <tt>nano</tt> editor.

Next, if you like, take a look through the <tt>ebuild</tt> files in the overlay (and any patches), read the underlying sources, and satisfy yourself that everything is benign. It is good hygiene to do this, particularly prior to using a third-party overlay for the first time.

Now, with that setup done, <span id="install_showem">we can now install the <tt>showem</tt> tool using the standard <tt>emerge</tt> command. Issue:

And that's <tt>showem</tt> installed! We're about to do quite a lot of building, so it'll be useful to <span id="second_virtual_console">set up a second virtual console, from which we can view the progress of <tt>emerge</tt> using our new script. Assuming you are running <tt>screen</tt> (as discussed earlier), press then  to start a new console. Then in that new console (which is back outside the <tt>chroot</tt>, to begin with) enter:

followed by

Now hit then  to get back to the original console, and continue.

<span id="bootstrapping_base_system">Bootstrapping the Base System (Optional but Recommended)
We are about to perform the bootstrap proper! In the below, we will follow the Gentoo FAQ's advice on how to recreate a 'stage 1' bootstrap in the modern age ^-^. (However, if you are content to rely on the shipped binaries, click here to skip this step.)

Note that here, bootstrapping refers to the process of:
 * 1) building (from source) the standard toolchain (GCC compiler, linker, assembler, C library and a number of other items), i.e., the components necessary to build the other software and libraries that make up Gentoo's <tt>@world</tt> package set; and then
 * 2) using that newly constructed toolchain to rebuild everything in the <tt>@world</tt> package set, from source.

And the point of this whole exercise? Simply put, it is to ensure the integrity of the end binaries, as far as we can. To reiterate: we will first build the core tools from source, using (by necessity) the existing toolchain binaries that are shipped as part of the minimal 'stage 3' filesystem (which we're currently <tt>chroot</tt>-ed inside of). We then use this newly built toolchain to rebuild (from source) all the shipped software in the stage 3 tarball (plus that which we've explicitly emerged, such as ). Once this has been done, we verify that all libraries and binaries (apart from the kernel, which we haven't built yet) have indeed been regenerated, and that no old binaries or libraries are still lying around.

In Gentoo parlance, people speak of three 'stages' of bootstrapping (and their corresponding file system tarballs):
 * 1) Stage 1: When starting from a stage 1 tarball, the base toolchain (GCC, standard C libary etc.) must be built using the existing (binary) host system toolchain, under direction of the  script. This yields a:
 * 2) Stage 2 system. Here, we still need to <tt>emerge</tt> (build) the core <tt>@world</tt> package set, using our new toolchain. This yields a:
 * 3) Stage 3 system, in which the toolchain has been bootstrapped, and the important system binaries and libraries have been compiled using it. A tarball of such a stage 3 system's directories is now provided as a default part of the Gentoo distribution (stage 1 and stage 2 tarballs are not available to end-users anymore).

Although we have already downloaded a stage 3 tarball, we're going to pretend we haven't, and instead build up from stage 1.

<span id="bootstrap_from_1_to_2">Gentoo Bootstrap Remix: Progressing from Stage 1 to Stage 2
Right, let's get going. First, we'll need to build ourselves a toolchain! We'll switch to the correct directory, and then do a dummy run to see what the supplied <tt>bootstrap.sh</tt> script wants to do:

then

Start the build for real this time:

This will take a while! You can <span id=use_showem>now switch to the second <tt>screen</tt> console we prepared earlier, and see what is going on (as the files download, build etc.). Hit then  to switch to the second console (you can do this while the bootstrap is running), and issue:

You can now switch back and forward between the two windows as you like (using then  to cycle forwards, and  then  to cycle backwards - identical in function here where we only have two windows active), which should give you a good overview of the bootstrap process as it progresses.

Assuming the <tt>bootstrap.sh</tt> script completes successfully, you next need to check your GCC configuration (since we just rebuilt the compiler, and possibly upgraded it too). Switch back to the first console (hit then ), and issue:

(That's an 'ell' by the way, not a one!) If the output tells you your current config is invalid, enter:

(That's a one this time, not an 'ell'!)

Next, we'll run the <tt>bootstrap.sh</tt> script again, to ensure that everything in the toolchain (including early dependencies, such as and ) have been rebuilt using the new compiler, and are then themselves used in a rebuild of that compiler:

Now, just as above, you can switch back and forth between the two <tt>screen</tt> consoles, to review the progress.

Once the toolchain bootstrap has completed (for the second time), we're ready to rebuild all the other binaries using it. You shouldn't need to reset the GCC profile this time around, but just to be on the safe side, double-check it is valid:

And finally, revert back to the root directory:

<span id="bootstrap_from_2_to_3">Gentoo Bootstrap Remix: Progressing from Stage 2 to Stage 3
We now need to build everything in the <tt>@world</tt> package set, using our shiny new toolchain.

Before we start, we need to <span id="create_build_timestamp">write a null 'timestamp' file, which we'll use as a marker when checking later that all our executables and libraries have indeed been rebuilt. Issue:

Right, with that done, we are ready to build everything in the <tt>@world</tt> package set, including all the libraries upon which they depend, as follows:

Here's what those <tt>emerge</tt> parameters mean (see also the emerge manpage):

After some time, the build will hopefully complete successfully (if it does not, please see the troubleshooting section, below).

Assuming you modified your file earlier, then Portage will probably prompt you with:

in the output from the full <tt>emerge</tt> run. This is <span id="using_dispatch_conf">easily dealt with, using the <tt>dispatch-conf</tt> tool (and it's good hygiene to run this anyway, after a large build session). This utility exploits the fact that Portage does not install packages directly to the filesystem, but rather uses a sandbox approach. As a result, it will not directly overwrite files in directories protected by the <tt>CONFIG_PROTECT</tt> variable (includes most files under in a standard setup, you can use <tt>emerge --info | grep CONFIG_PROTECT</tt> to see the full details). Instead, it writes the new configs to .cfg0000_ files. <tt>dispatch-conf</tt> lets you step through each of these conflicting files in turn, shows you a <tt>diff</tt> between your current version and what the install wanted to write, and asks you what you want to do.

Enter:

Then follow the prompts to review each proposed change. Assuming only the needs modifying (should be the case at this stage), then you simply need to press. This means (rather non-obviously) 'zap-new' - or in other words keep the version of the file that was there before the <tt>emerge</tt> was initiated. Once this has been done, <tt>dispatch-conf</tt> should exit automatically.

If everything worked fine, then you can skip ahead to the next section; otherwise, here are a few tips on how to rectify any problems that may have occurred.

<span id="troubleshooting_failed_build">Troubleshooting a Failed Build
Generally, <tt>emerge</tt>s, even of the full <tt>@world</tt> set, go without a hitch. However, if you elected earlier to use the '~amd64' (aka 'testing') phase ebuilds, issues will sometimes pop up. For example, here's a problem that happened during the writing of this guide, when attempting the stage 2 -> stage 3 <tt>emerge</tt> (I was using the testing branch then, since GNOME 3 had not been stabilized at the time - nevertheless, it's an instructive example):



In this case, a build failure occurred while emerging the package. Here are some steps you can take if this kind of thing happens to you (this is not an in-depth guide - for that see this section from the Gentoo handbook):
 * Sometimes, intermittent issues occur due to the very parallel build architecture we have specified. These issues will often resolve themselves if the build is recommenced. Fortunately, there's a flag for that. Issue and <tt>emerge</tt> will try to pick up where it left off, with all the same settings. Alternatively, you can try to resume the <tt>emerge</tt> with parallel <tt>make</tt> temporarily switched off; to do so, issue  If that works OK, great, you're done. If not, proceed to the next step.
 * Since you are (at the moment) running on a fairly 'vanilla' system, it's likely that others will have seen the build problem you're experiencing, and have already reported it. Gentoo has a pretty quick turnaround for fixes, which are posted to the Portage (ebuild) tree when signed off. Therefore, the next-lowest-overhead way to fix a build problem is simply to wait a little while (if it's a major issue, you shouldn't have to wait long), then resync Portage, and try the <tt>emerge</tt> again (NB, but not using <tt>--resume</tt> here, since we've sync'd in between): Hopefully, that fixed your problem. If not (or you don't want to wait), keep going...
 * Look at the full build log. As the build has failed, this will still be in somewhere - indeed, in this example, the actual file is at . Have a read. Having done so, try searching for any appropriate-sounding errors using Google (or, if you want to use an anyonymized wrapper around the Google search engine, Startpage). For example, here the text is quite specific and suggestive. A Google search for "<tt>Gentoo ~amd64 emerge media-libs/mesa-9.2.5 Makefile:603: recipe for target 'all-recursive' failed</tt>" yields a relevant hit: a post on the Gentoo discussion forums: "mesa-9.2.5 fails, egl_gallium/llvm related". A quick perusal shows us we have found the correct problem, and that it is related to, a problem with the package . In short, it appears that there is a problem with the default-use-flag build of , it's just that it shows itself in the build of . You can now try to:
 * 1) Work around the problem, per the information you have found. For example, in this case, you could try appending <tt>~sys-devel/llvm-3.4 -ncurses</tt> to the  file (see this earlier discussion of atoms and EFI_Gentoo_End_to_End_Install/Installing_the_Gentoo_Stage_3_Files if you're unclear as to what this means) and issuing the full <tt>@world</tt> <tt>emerge</tt> again. Just don't forget that you have that you have made this change - and do try to come back and see if you can revert the fix later (were a new ebuild for LLVM to be released, for example)!
 * 2) Try using an earlier version of the affected package (this is, in fact, the approach I adopted at the time, although it was rendered moot by the stabilization of GNOME 3). You can do this using the  file (discussed earlier), which tells Portage the versions of software not to allow. Here, we could try preventing Portage from using the 3.4 version of, by adding <tt>=sys-devel/llvm-3.4</tt> to  (forcing Portage to reject it), then re-emerging. In this case, from the online package information grid for LLVM, we can see this will drop us back to version 3.3-r3 (remember, I was on the testing branch). This approach generally works, but be warned - you may end up rolling back quite a lot of other packages when you mask in this way. Furthermore, if you adopt this approach, always be sure only to mask the specific version that has the problem (use '=' at the start of the  line, per our earlier discussion of atoms), as this means when a new (hopefully, fixed) ebuild is released (with a higher version number), Portage will automatically attempt to try to use that instead.

Finally, here are three miscellaneous hints that you may find useful:
 * 1) Check that your <tt>perl</tt> libraries are up-to-date. This will not always be handled properly by Portage, but you can use <tt>perl-cleaner</tt> to bring things back in line. Issue:
 * 2) Ditto for <tt>python</tt>. Issue:  and then re-attempt the failed <tt>emerge</tt>.
 * 3) Ensure that <tt>ruby</tt> is set to the latest version, if it is present in your system (if the following <tt>eselect ruby list</tt> command returns an error, you do not). Certain things like  require an up-to-date version. Issue:  to see a list of available versions, and then:  to choose one.
 * Then re-attempt the failed <tt>emerge</tt>.

<span id="verifying_bootstrap">Verifying the Bootstrap (Optional but Recommended)
Congratulations - by this point you have performed a 'stage 1' bootstrap, and so all the (binary) applications and libraries within the current <tt>chroot</tt> should have been recompiled. Just to make sure that nothing has sneaked sneaked through (i.e., an app or library somehow included in the stage 3 tarfile we downloaded earlier, but not present in the deep transitive closure of the <tt>@world</tt> package set), we'll check the datestamps of all relevant components against the we wrote before commencing the <tt>emerge</tt>. We aren't worried about any text executables (such as scripts) since, in principle, these can be audited to ensure they are not malicious.

Issue:

This command finds all executable files (except beneath the EFI mountpoint at ; which will host a FAT filesystem - in which all files appear executable - when mounted (it currently shouldn't be mounted though, assuming you've been following the tutorial flow, but the <tt>prune</tt> is there for robustness)), which aren't newer than the checkpoint file, then tests their file type using <tt>file</tt>, and prints them if they are not recognized as some form of text file (script, source code etc.).

At the time of writing on the author's machine, running this command yields only one file,, which (as the name suggests) is just a bzip2 archive containing a script. If this happens to you, you may wish to unset the executable bits on this:

So much for executable binaries - what about libraries (shared or static)? On Gentoo, we'll have already checked most shared ('.so') libraries, since they have their execute bits set. However, this is not true for all flavours of Linux, and furthermore, static ('.a') libraries (and '.o' object files) do not have the execute bits set. Therefore, to be double-sure, we'll run another test (this may take some time to complete):

This looks for any files which aren't newer than our checkpoint file, and which are recognized by <tt>file</tt> as ELF or ar archives; it may take some time to complete (as there are a lot of files to check). It should produce no output (as all files of this type should have been recreated, if our bootstrap was complete). If so, and the previous check also worked, then congratulations, you have fully rebuilt all executable binaries and libraries on your machine!

<span id="choose_profile">Setting the Target Profile, and Updating
With that done, we can now <span id="set_profile">set our Portage profile to the version we actually want to use going forward (we avoided doing so earlier, to minimize the scope of the bootstrap).

Issue:

The current (default) profile is shown marked with an asterisk. However, the most correct profile for our purposes is the 'desktop' variant that cites GNOME and <tt>systemd</tt>. In the above list, this is number 5 (but your output may differ, so be sure to check).

To switch, issue:

Having done this, we must update <tt>@world</tt> set, to pull in (and build from source) any new packages required, and to reflect any USE flag changes to existing packages (by rebuilding them as necessary).

Issue:

Here's <span id="world_update_params">what those <tt>emerge</tt> parameters mean (see also the emerge manpage):

Whether or not <tt>emerge</tt> warns you that you have configuration files that need updating as a result of the above update, it is good hygiene to run:

Then follow the prompts to review each proposed change, if any.

<span id="next_steps">Next Steps
Having successfully built the <tt>@world</tt> set for our new profile, we can now turn our attention to the heart of the system: the Linux kernel itself. Click here to go to the next chapter, "Configuring and Building the Kernel".

Acknowledgements
We would like to thank the following authors and editors for their contributions to this guide: