Binary package guide
Next to the usual support for ebuilds, portage supports building and installing binary packages. This guide explains how to create them, how to install them and how to setup a binary package server.
- 1 Introduction
- 2 Creating binary packages
- 3 Setting up a binary package host
- 4 Using binary packages
- 5 Maintaining binary packages
- 6 Advanced topics
There are many reasons why some administrators like using binary package installations in Gentoo.
- First of all, it allows administrators to keep similar systems updated. Having to compile everything from source can become time consuming at times. Maintaining several similar systems, possibly some of them rather slow, can be much easier if only one system has to compile everything from source and the other systems reuse the binary packages.
- A second reason is to do safe updates. For mission critical systems it is important to stay usable as much as possible. This can be done by a staging server that performs all updates first and once the staging server is in a good state updates can be applied to the critical system. A variant of this approach is to do the updates in a chroot on the same system and use the binaries created there on the real system.
- A third reason is as a backup. Sometimes binary package are the only way of recovering a broken system (i.e. broken compiler). Having them around either on a binary package server or locally can be of great help.
- Finally, it also supports updating very old systems. The task of updating very old systems can be greatly eased using binary packages. It is usually easier to install binary packages as those don't require the build time dependencies to be installed / updated and the failures in the build process for them can be avoided.
In this guide, we focus on
- how to create binary packages,
- how to distribute these to the clients,
- how to use binary packages, and
- how to maintain the binary packages
We end with a few more advanced topics on dealing with binary packages.
Creating binary packages
There are three main methods for creating binary packages:
- After a regular installation, using
- Explicitly during an emerge operation by using the
- Automatically through the use of the
All three methods will create a binary package in the directory pointed to by the
PKGDIR variable (which defaults to /usr/portage/packages).
quickpkg application takes one or more dependency atoms (or package sets) and creates binary packages for all installed packages that match the atom.
For instance, to create binary packages of all installed GCC versions:
To create binary packages of all installed packages, use the
There is a caveat with this method though: it relies on the installed files, which can be a problem in case of configuration files. Administrators often change configuration files after installing software. Because this could leak out important or perhaps even confidential data into the packages,
quickpkg by default does not include configuration files that are protected through the CONFIG_PROTECT method. To include those as well, use the
Using the --buildpkg emerge option
When installing software using
emerge, portage can be asked to create binary packages as well by using
It is also possible to ask portage to only create a binary package but not to install the software on the live system. For this, the
--buildpkgonly (-B) option can be used:
The latter approach however requires that all build time dependencies are already installed.
Automatically using buildpkg feature
The most common way is to automatically create binary packages whenever a package is installed by
emerge. This is done through the buildpkg feature, which can be set in /etc/portage/make.conf like so:
Every time portage installs software, it will create a binary package as well.
Excluding creation of some packages
It is possible to tell portage not to create binary packages for a select few packages or categories. This is done through the
--buildpkg-exclude option to emerge:
This could be used for packages that have little to no benefit in having a binary package available. Examples would be the Linux kernel source packages or upstream binary packages (those ending with -bin like www-client/firefox-bin).
Setting up a binary package host
Portage supports a number of protocols for downloading binary packages: FTP, FTPS, HTTP, HTTPS and SSH. This leaves room for many possible binary package host implementations.
There is, however, no out-of-the-box method provided by portage for distributing binary packages. Depending on the requested setup, additional software will need to be installed.
Web based binary package host
A common approach for distributing binary packages is to create a web based binary package host.
Use a web server (such as www-servers/lighttpd) and configure it to provide read access to the
Then, on the clients, configure the
PORTAGE_BINHOST variable accordingly:
SSH binary package host
To provide a more authenticated approach for binary packages, one can consider using SSH.
When using SSH, it is possible to use the portage Linux user's SSH key (without passphraze as the installations need to happen in the background) to connect to a remote binary package host.
To accomplish this, make sure that the portage user's SSH key is allowed on the server:
PORTAGE_BINHOST could then look like so:
When using binary packages on an internal network, it might be easier to just export the packages through NFS and mount it on the clients.
The /etc/exports file could look like so:
On the clients, the location can then be mounted. An example /etc/fstab entry would look like so:
Using binary packages
For binary packages to be usable on other systems they must fulfill some requirements.
- The client and server architecture and CHOST must match.
CXXFLAGSthat were used to build the binary packages must be compatible with all clients.
- USE flags for processor specific features (like MMX, SSE,...) have to be carefully selected; all clients need to support them.
Next to these, portage will check if the binary package is built using the same USE flags as expected on the client. If a package is built with a different USE flag combination, portage will either ignore the binary package (and use source-based build) or fail, depending on the options passed on to
emerge (see Installing binary packages).
On clients, a few configuration changes are needed in order for the binary packages to be used.
Installing binary packages
There are a few options that can be passed on to the
emerge command that inform portage about using binary packages.
- Try to use the binary package(s) in the locally available packages directory. Useful when using NFS-mounted binary package hosts. If the binary package is not found, a regular (source-based) installation will be done.
- Similar to
--usepkg (-k)but fail if the binary package cannot be found.
- Download the binary package from a remote binary package host. If the binary package is not found, a regular (source-based) installation will be done.
- Similar to
--getbinpkg (-g)but fail if the binary package cannot be downloaded.
In order to automatically use binary package installations, the appropriate option can be added to the
There is a portage feature that automatically implements the equivalent of
--getbinpkg (-g) without the need for updating the
EMERGE_DEFAULT_OPTS variable: getbinpkg.
Pulling packages from a binary package host
When using a binary package host, clients need to have the
PORTAGE_BINHOST variable set. Otherwise the client will not know where the binary packages are stored (and how to retrieve them).
PORTAGE_BINHOST variable uses a space-separated list of URIs. This allows administrators to use several binary package servers simultaneously. The URI must always point to the directory in which the Packages file resides.
Reinstalling modified binary packages
--rebuilt-binaries option to
emerge will reinstall every binary that has been rebuild since the package was installed. This is useful in case rebuilding tools like
python-updater are run on the binary package server.
A related option is
--rebuilt-binaries-timestamp. It causes emerge not to consider binary packages for a re-install if those binary packages have been built before the given time stamp. This is useful to avoid re-installing all packages, if the binary package server had to be rebuild from scratch but
--rebuilt-binaries is used otherwise.
Additional client settings
Next to the getbinpkg feature, portage also listens to the binpkg-logs feature. This one controls if log files for successful binary package installations should be kept. It is only relevant if
PORT_LOGDIR is set and is enabled by default.
Similar to excluding binary packages for a certain set of packages or categories, clients can be configured to exclude binary package installations for a certain set of packages or categories.
To accomplish this, use the
Maintaining binary packages
Exporting and distributing the binary packages will lead to useless storage consumption if the binary package list is not actively maintained.
Removing outdated binary packages
In the app-portage/gentoolkit package an application called
eclean is provided. It allows for maintaining portage-related variable files, such as downloaded source code files, but also binary packages.
The following command will remove all binary packages that have no corresponding ebuild:
For more details please read the Eclean article.
Another tool that can be used is the
qpkg tool from the app-portage/portage-utils. However, this tool is a bit less configurable.
To clean up unused binary packages (in the sense of used by the server on which the binary packages are stored):
Maintaining the Packages file
Inside the packages directory, a file called Packages exists. This file acts as a cache for the metadata of all binary packages in the packages directory. The file is updated whenever portage adds a binary package to the directory. Similarly,
eclean updates it when it removes binary packages.
If for some reason binary packages are simply deleted or copied into the packages directory, or the Packages file gets corrupted or deleted, then it needs to be recreated. This is done using
Creating snapshots of the packages directory
When deploying binary packages for a large number of client systems it might become worthwhile to create snapshots of the packages directory. The client systems then don't use the packages directory directly but use binary packages from the snapshot.
Snapshots can be created using the /usr/lib64/portage/bin/binhost-snapshot tool. It takes four arguments,
- a source directory (the path to the packages directory),
- a target directory (that must not exist),
- a URI, and
- a binary package server directory.
The files from the package directory are copied to the target directory. A Packages file is then created inside the binary package server directory (fourth argument) with the provided URI.
Client systems need to use an URI that points to the binary package server directory. From there they will be redirected to the URI that was given to
binhost-snapshot. This URI has to refer to the target directory.
Understanding the binary package format
Binary packages created by portage have the file name ending tbz2. These files consist of two parts:
- an .tar.bz2 archive containing the files that will be installed on the system, and
- an xpak archive containing the metadata, the ebuild and the environment file.
man xpak for a description of the format.
In app-portage/portage-utils some tools exists that are able to split or create tbz2 and xpak files.
The following command will split the tbz2 into a .tar.bz2 and an .xpak file:
The xpak file can be examined using
To list the contents:
The next command will extract a file called USE which contains the enabled use flags for this package:
The PKGDIR layout
The currently used format version 2 has the following layout:
The Packages is the major improvement (and also the trigger for portage to know that the binary package directory uses version 2) over the first binary package directory layout (version 1). In version 1, all binary packages were also hosted inside a single directory (called All/) and the category directories only had symbolic links to the binary packages inside the All/ directory.
Unpacking with quickunpkg
Zoobab wrote a simple shell tool named quickunpkg to quickly unpack tbz2 files.