Binary package guide

Next to the usual support for ebuilds, portage supports building and installing binary packages. This guide explains how to create them, how to install them and how to setup a binary package server.

Preliminaries
Terms used in this guide:
 * binhost or binary package server
 * The system that creates and redistributes binary packages.


 * client system
 * The system that downloads and installs binary packages.

Tools used in this guide: If not explicitly stated all referenced tools are part of. This guide was written with sys-apps/portage-2.1.10.28 in mind.

The guide is structured as follows. The first part outlines some use cases for binary packages and binary package servers. The second part describes how to create binary packages on a Gentoo system. The third part describes how to redistribute the binary packages. The fourth part covers the configuration and typical workflow for client systems. The last part covers advanced topics.

Keeping similar systems updated
Having to compile everything from source can become time consuming at times. Maintaining several similar systems, possibly some of them rather slow, can be much easier if only one system has to compile everything from source and the other systems reuse the binary packages.

Doing safe updates
For mission critical systems it is important to stay usable as much as possible. This can be done by a staging server that performs all updates first and once the staging server is in a good state updates can be applied to the critical system.

A variant of this approach is to do the updates in a chroot on the same system and use the binaries created there on the real system.

As a backup
Sometimes binary package are the only way of recovering a broken system (i.e. broken compiler). Having them around either on a binary package server or locally can be of great help.

Updating very old systems
The task of updating very old systems can be greatly eased using binary packages. It is usually easier to install binary packages as those don't require the build time dependencies to be installed / updated and the failures in the build process for them can be avoided.

Creating binary packages
Depending on what you want to use the binary packages for, there are several ways to configure emerge.

The most common way is to create binary packages whenever a package is installed using an ebuild. To do so, enable the FEATURE 'buildpkg' (see man make.conf).

To create binary packages on demand use the --buildpkg (-b) option for emerge. All packages installed from ebuilds with this command will have a binary package created. There also exists the --buildpkgonly (-B) which only create binary packages, but doesn't install the package. This option is not recommended, because it requires that all build time dependencies are already installed. If this requirement is not met, packages will fail to build.

A third option is to create binary packages from already installed packages using 'quickpkg'. 'quickpkg' takes one or more dependency atoms or package set (for portage versions supporting this) as argument and creates binary packages for all installed packages matching these atoms.

One caveat with this method is that it relies on the installed files, which is especially a problem for config files, which frequently are modified compared to what the package originally installed. Files which are protected by CONFIG_PROTECT (see man portage) aren't included by default. To include them use the --include-config or --include-unmodified-config option (see man quickpkg).

How to not create binary packages
Sometimes it is desirable to not build binary packages for some packages. The most common reason being that the package does not perform noticeable amounts of work during building, like kernel sources or packages that only install an upstream provided binary package. The --buildpkg-exclude option does exactly this.

Example:

This works independently of how building of binary packages was enabled.

Configuring paths
By default binary packages are stored in the path defined as PKGDIR in /etc/portage/make.conf. It defaults to PORTDIR/packages, which itself defaults to /usr/portage/packages.

Other configuration variables
For binary packages to be usable on other systems they must fulfill some requirements.
 * First, architecture and CHOST must match.
 * Second, the C(XX)FLAGS have to be chosen in a way that both the binary package server and the client systems can run the binaries.
 * Third, USE flags for processor specific features like MMX, SSE,... have to be chosen in a way that all systems have the enabled features, or the packages using these features have to be rebuilt on the client systems from source.

Setting up a binary package host
Portage supports the following protocols for downloading binary packages:
 * ftp
 * ftps
 * http
 * https
 * ssh

Portage itself does not act as server that redistributes the binary packages. Instead you have to install a server that provides one of the protocols above.

Example using to setup an http server.

Pointing your browser to http://127.0.0.1/Packages should download the Packages file in PKGDIR (assuming you already have binary packages).

Example using to setup an ssh server. (There may be a better way to do this, but the process of using SSH for this is a little hairy.)

Maintaining a binary package server
This chapter covers issues beyond keeping the server updated.

Removing outdated binary packages
provides a tool called 'eclean' that does exactly this.

The following command will remove all binary packages that have no corresponding ebuild.

For more details please read the Eclean article.

A less customizable option is 'qpkg' from.

Recreating / updating the Packages file
PKGDIR contains a file called Packages. This file is a cache for the metadata of all binary packages in PKGDIR. The file is updated whenever portage adds a binary package to PKGDIR. Similarly, eclean updates it when it removes binary packages.

If for some reason binary packages are simply deleted or copied into PKGDIR or the Packages get corrupted or deleted, then it needs to be updated / recreated. This is done using 'emaint.

PKGDIR
The PKGDIR (set in make.conf) variable decides where binary packages are stored. This directory is always required when using binary packages as the binary packages downloaded from the binary package server will be stored there too.

PORTAGE_BINHOST
The PORTAGE_BINHOST (set in make.conf) is a space separated URI list. This way it is possible to use several binary package servers simultaneously. The URI must always point to the place where the Packages file resides on the binary package server.

The support for multiple binary package servers is somewhat incomplete. If several servers serve a binary package for the same package version, then only the first one will be considered. This can be problematic when these binary packages differ in their USE configuration and the USE configuration of a later binary package would match the systems configuration.

FEATURES
The 'getbinpkg' feature tells portage to always use the configured binary package server. Enabling this feature is equivalent to the emerge option -g (see below).

The 'binpkg-logs' controls if log files for successful binary package installs should be kept. It is only relevant if PORT_LOGDIR is set. It is enabled by default.

emerge options
There are four options that tell portage to use binary packages. Some restrict portage to local binary packages already present in PKGDIR, some also allow fetching them from binary package servers. They also decide if only binary packages should be considered or both ebuilds and binary packages should be used.

For local binary packages and ebuilds use: --usepkg (-k)

For local binary packages only: --usepkgonly (-K)

For local and remote binary packages and ebuilds: --getbinpkg (-g)

For local and remote binary packages only: --getbinpkgonly (-G)

The options ignoring ebuilds require binary packages to be present for every needed package or 'emerge' will report an error. These options are useful to sync a system against a binary package server while avoiding any compilation on the client system.

Excluding binary packages
Sometimes (when using -k or -g) it is desirable to use an ebuild over a binary package. This can be accomplished with the --usepkg-exclude option. All binary packages given with this option will be ignored.

Example:

Reinstalling changed binary packages
The --rebuilt-binaries option will reinstall every binary that has been rebuild since the package was installed. This is useful in case rebuilding tools like revdep-rebuild or python-updater are run on the binary package server.

A related option is --rebuilt-binaries-timestamp. It causes emerge not to consider binary packages for a re-install if those binary packages have been built before the given time stamp. This is useful to avoid re-installing all packages, if the binary package server had to be rebuild from scratch.

Creating snapshots of PKGDIR
When deploying binary packages for a large number of client systems it might become worthwhile to create snapshots of PKGDIR. The client systems then don't use PKGDIR directly but use binary packages from the snapshot.

Snapshots can be created using the /usr/lib64/portage/bin/binhost-snapshot tool.

It takes four arguments, a source directory (PKGDIR), a target directory that must not exist, a URI and a binary package server directory.

What then happens is that PKGDIR gets copied to the target dir. In the binary package server directory a packages file is created which contains the provided URI.

Client systems need to use an URI that points to the binary package server directory. From there they will be redirected to the URI that was given to binhost-snapshot. This URI has to refer to the target directory.

The binary package format
Binary packages created by portage have the file name ending tbz2. These files consist of two parts an .tar.bz2 archive containing the files that will be installed on the system and an xpak archive containing the metadata, the ebuild and the environment file. See 'man xpak' for a description of the format.

Taking a tbz2 apart
In some tools exists, to that are able to split or create tbz2 and xpak files.

The following command will split the tbz2 into a tar.bz2 and an xpak file.

The xpak file can be examined using qxpak. List the contents:

The next command will extract a file called USE which contains the enabled use flags for this package.

The PKGDIR layout
The currently used format version 2 has the following layout.

PKGDIR layout version 2

Binary package server clients are pointed to PKGDIR.

As described earlier the Packages file is a metadata cache for the contained binary packages. This greatly improves dependency resolution speed over format 1.

The old format version 1, which should no longer be used has the following layout.

PKGDIR layout version 1

Portage still supports this format and falls back to it if the Packages file does not exist.

Quickunpkg
Zoobab wrote a simple shell tool named quickunpkg to quickly unpack tbz2 files: https://github.com/zoobab/quickunpkg