Binary package guide

From Gentoo Wiki
Jump to:navigation Jump to:search
This page contains changes which are not marked for translation.
This guide uses the PORTAGE_BINHOST variable to configure binhosts. There now exists binrepos.conf, which the Portage man page says is intended to be used as a replacement for PORTAGE_BINHOST.

Next to the usual support for source-based ebuilds, Portage also supports building and installing binary packages. This guide explains how to create them, install them, and how to setup a binary package server.

There are many reasons why some system administrators like using binary packages for software installations on Gentoo:

  • Save time when keeping similar systems updated. Having to compile everything from source can become time consuming. Maintaining several similar systems, possibly some of them with older hardware, can be much easier if only one system has to compile everything from source and the other systems use the binary packages.
  • Do safe updates. For mission-critical systems in production it is important to stay usable as much as possible. This can be done by a staging server that performs all updates first to itself. Once the staging server is in a good state the updates can then be applied to the critical systems. A variant of this approach is to do the updates in a chroot on the same system and use the binaries created there on the real system.
  • As a backup. Often binary packages are the only way of recovering a broken system (i.e. broken compiler). Having pre-compiled binaries around either on a binary package server or locally can be of great help in case of a broken toolchain.
  • It aids in updating very old systems. The task of updating very old systems can be greatly eased using binary packages. It is usually helpful to install binary packages on old systems because they do not require build time dependencies to be installed/updated. Binaries packages also avoid failures in build processes since they are pre-compiled.

This guide will focus on the following topics:

  • Creating binary packages.
  • Distributing the packages to clients.
  • Implementing binary packages.
  • Maintaining the binary packages.

Near the end, a few more advanced topics on dealing with binary packages will be covered.

All tools used in this guide are part of sys-apps/portage, unless otherwise stated.

Binary package formats

Starting with portage version 3.0.31, a new binary package format (GPKG) support has been added. The new format was completely redesigned to solve issues with the old design and to include new features, therefore no backward compatibility is provided. If users need to create binary packages for the system using older versions of Portage, please keep using the current XPAK format.

Motivation for the new GPKG format can be found in GLEP 78: Gentoo binary package container format. See also bug #672672 and bug #820578.

To use new GPKG format, set the BINPKG_FORMAT value in /etc/portage/make.conf. The default value is xpak.

FILE /etc/portage/make.confSpecify binary package format

This guide applies to both formats. See the understanding the binary package format section for more information on the binary package formats.

Creating binary packages

There are three main methods for creating binary packages:

  1. After a regular installation, using the quickpkg application.
  2. Explicitly during an emerge operation by using the --buildpkg (-b) option.
  3. Automatically through the use of the buildpkg (build binary packages for all packages) or buildsyspkg (build binary packages only for the system set) values in Portage's FEATURES variable.

All three methods will create a binary package in the directory pointed to by the PKGDIR variable (which defaults to /var/cache/binpkgs).

Using quickpkg

The quickpkg application (included in Portage) takes one or more dependency atoms (or package sets) and creates binary packages for all installed packages that match that atom.

There is a caveat with this method: it relies on the installed files, which can be a problem in case of configuration files. Administrators often change configuration files after installing software. Because this could leak out important (perhaps even confidential) data into the packages, quickpkg by default does not include configuration files that are protected through the CONFIG_PROTECT method. To force inclusion of configuration files, use the --include-config or --include-unmodified-config options.

For instance, to create binary packages of all installed GCC versions:

root #quickpkg sys-devel/gcc

To create binary packages for the system set:

root #quickpkg @system

To create binary packages of all installed packages on the system, use the * glob:

root #quickpkg "*/*"

Using --buildpkg as an emerge option

When installing software using emerge, Portage can be asked to create binary packages by using --buildpkg (-b) option:

root #emerge --ask --buildpkg sys-devel/gcc

It is also possible to ask Portage to only create a binary package but not to install the software on the live system. For this, the --buildpkgonly (-B) option can be used:

root #emerge --ask --buildpkgonly sys-devel/gcc

The latter approach however requires all build time dependencies to be previously installed.

Implementing buildpkg as a Portage feature

The most common way to automatically create binary packages whenever a package is installed by Portage is to use the buildpkg feature, which can be set in /etc/portage/make.conf like so:

FILE /etc/portage/make.confEnabling Portage's buildpkg feature

With this feature enabled, every time Portage installs software it will create a binary package as well.

Excluding creation of some packages

It is possible to tell Portage not to create binary packages for a select few packages or categories. This is done by passing the --buildpkg-exclude option to emerge:

root #emerge -uDN @world --buildpkg --buildpkg-exclude "virtual/* sys-kernel/*-sources"

This could be used for packages that have little to no benefit in having a binary package available. Examples would be the Linux kernel source packages or upstream binary packages (those ending with -bin like www-client/firefox-bin).

Binary package compression formats

It is possible to use a specific compression type on binary packages. Currently, the following formats are supported: bzip2, gzip, lz4, lzip, lzop, xz, and zstd. Defaults to zstd. Review man make.conf and search for BINPKG_COMPRESS for the most up-to-date information.

The compression format can be specified via make.conf.

FILE /etc/portage/make.confSpecify binary package compression format

Note that the compression type used might require extra dependencies to be installed, for example, in this case app-arch/lz4.

Binary package OpenGPG signing

OpenGPG signing is only available in GPKG binpkg format.

GPG signature allows users to check the creator and integrity of a binary package, and to perform trust management based on keys. The binary package signing feature is disabled by default, to use it users need to enable the binpkg-signing feature. Note that whether this feature is enabled does not affect the signature verification feature.

FILE /etc/portage/make.confEnabling Portage's binpkg-signing feature

Users also need to set the BINPKG_GPG_SIGNING_GPG_HOME and BINPKG_GPG_SIGNING_KEY variables for Portage to find the signing key.

FILE /etc/portage/make.confConfiguring Portage's signing key

Portage will only try to unlock the GPG private key at the beginning. If the user's key will expire over time, then consider enabling gpg-keepalive to prevent signing failures.

FILE /etc/portage/make.confEnabling Portage's gpg-keepalive feature

Setting up a binary package host

Portage supports a number of protocols for downloading binary packages: FTP, FTPS, HTTP, HTTPS, and SSH/SFTP. This leaves room for many possible binary package host implementations.

There is, however, no "out-of-the-box" method provided by Portage for distributing binary packages. Depending on the desired setup additional software will need to be installed.

Web based binary package host

A common approach for distributing binary packages is to create a web-based binary package host.

Use a web server such as lighttpd (www-servers/lighttpd) and configure it to provide read access to /etc/portage/make.conf's PKGDIR location.

FILE /etc/lighttpd/lighttpd.conflighttpd configuration example
# add this to the end of the standard configuration
server.dir-listing = "enable"
server.modules += ( "mod_alias" )
alias.url = ( "/packages" => "/var/cache/binpkgs/" )

Then, on the client systems, configure the PORTAGE_BINHOST variable accordingly:

FILE /etc/portage/make.confUsing a web-based binary package host

SSH binary package host

To provide an authenticated approach for binary package mirrors, Portage can be configured to use the SSH protocol to access binary packages.

When using SSH, it is possible to use the root Linux user's SSH key (without passphrase as the installations need to happen in the background) to connect to a remote binary package host.

To accomplish this, make sure that the root user's SSH key is allowed on the server. This will need to happen for each machine that will connect to the SSH capable binary host:

root #cat >> /home/binpkguser/.ssh/authorized_keys

The PORTAGE_BINHOST variable could then look like so:

FILE /etc/portage/make.confSetting PORTAGE_BINHOST for SSH access
Do not use the SSH configuration files found in ~/.ssh/config for setting ports or username. This location is ignored when Portage tries to rsync the packages back onto the client. Instead set all the options correctly in the PORTAGE_BINHOST variable.

NFS exported

When using binary packages on an internal network, it might be easier to export the packages through NFS and mount it on the clients.

The /etc/exports file could look like so:

FILE /etc/exportsExporting the packages directory
/var/cache/binpkgs   2001:db8:81::/48(ro,no_subtree_check,root_squash),no_subtree_check,root_squash)

On the clients, the location can then be mounted. An example /etc/fstab entry would look like so:

FILE /etc/fstabEntry for mounting the packages folder
binhost:/var/cache/binpkgs      /var/cache/binpkgs    nfs    defaults    0 0

The NFS share is mounted on the local filesystem, so there is no need to set PORTAGE_BINHOST or use the --getbinpkg option. Instead, follow the normal procedures for installing binary packages, remembering to point PKGDIR to the NFS share so that portage knows where to find the packages:

FILE /etc/portage/make.confSetting the package directory for portage
If PKGDIR is network-mounted, it may be advantageous to enable FEATURES="pkgdir-index-trusted". This feature disables checking the entire PKGDIR for added or removed packages and instead trusts the contents of the Packages file to be accurate. This significantly improves performance on high-latency networks.

Using binary packages

For binary packages to be usable on other systems they must fulfill some requirements:

  • The client and server architecture and CHOST must match.
  • The CFLAGS and CXXFLAGS variables used to build the binary packages must be compatible with all clients.
  • USE flags for processor specific instruction set features (like MMX, SSE, etc.) have to be carefully selected; all clients need to support them.
Portage can not validate if these requirements match. It is the responsibility of the system administrator to guard these settings.

The app-misc/resolve-march-native utility can be used to find a subset of CFLAGS that is supported by both the server and client(s). For example, the host might return:

user $resolve-march-native
 -march=skylake -mabm -mrtm --param=l1-cache-line-size=64 --param=l1-cache-size=32 --param=l2-cache-size=12288 

While the client might return:

user $resolve-march-native
 -march=ivybridge -mno-rdrnd --param=l1-cache-line-size=64 --param=l1-cache-size=32 --param=l2-cache-size=3072 

In this example CFLAGS could be set to -march=ivybridge -mno-rdrnd since -march=ivybridge is a full subset of -march=skylake. -mabm and -mrtm are not included as these are not supported but the client. However, -mno-rdrnd is included as the client does not support -mrdrnd. To find which -march's are subsets of others, check the gcc manual, if there is no suitable subset set e.g. -march=x86-64.

Optionally, it is also possible to set -mtune=some-arch or -mtune=native to tell gcc to tune code to a specific arch. In contrast to -march, the -mtune argument does not prevent code from being executed on other processors. For example, to compile code which is compatible with ivybridge and up but is tuned to run best on skylake set CFLAGS to -march=ivybridge -mtune=skylake. When -mtune is not set it defaults to whatever -march is set to.

When changing -march to a lower subset for using binary packages on a client, a full recompilation is required to make sure that all binaries are compatible with the client's processor, to save time packages that are not compiled with e.g. gcc/clang can be excluded:

user $emerge -e @world --exclude="acct-group/* acct-user/* virtual/* app-eselect/* sys-kernel/* sys-firmware/* dev-python/* dev-java/* dev-ruby/* dev-perl/* dev-lua/* dev-php/* dev-tex/* dev-texlive/* x11-themes/* */*-bin"

Similarly, app-portage/cpuid2cpuflags can be used to find a suitable subset of processor specific instruction set USE flags. For example, the host might return:

user $cpuid2cpuflags
 CPU_FLAGS_X86: aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt rdrand sse sse2 sse3 sse4_1 sse4_2 ssse3 

While the client might return:

user $cpuid2cpuflags
 CPU_FLAGS_X86: avx f16c mmx mmxext pclmul popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3 

In this example CPU_FLAGS_X86 can be set to avx f16c mmx mmxext pclmul popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3 in /etc/portage/make.conf because these flags are supported by both the client and the host

Next to these, Portage will check if the binary package is built using the same USE flags as expected on the client. If a package is built with a different USE flag combination, Portage will either ignore the binary package (and use source-based build) or fail, depending on the options passed to the emerge command upon invocation (see Installing binary packages).

On clients, a few configuration changes are needed in order for the binary packages to be used.

Installing binary packages

There are a few options that can be passed on to the emerge command that inform Portage about using binary packages:

Option Description
--usepkg (-k) Tries to use the binary package(s) in the locally available packages directory. Useful when using NFS or SSHFS mounted binary package hosts. If the binary packages are not found, a regular (source-based) installation will be performed.
--usepkgonly (-K) Similar to --usepkg (-k) but fail if the binary package cannot be found. This option is useful if only pre-built binary packages are to be used.
--getbinpkg (-g) Download the binary package(s) from a remote binary package host. If the binary packages are not found, a regular (source-based) installation will be performed.
--getbinpkgonly (-G) Similar to --getbinpkg (-g) but will fail if the binary package(s) cannot be downloaded. This option is useful if only pre-built binary packages are to be used.

In order to automatically use binary package installations, the appropriate option can be added to the EMERGE_DEFAULT_OPTS variable:

FILE /etc/portage/make.confAutomatically fetch binary packages and fail the package if not available

There is a Portage feature that automatically implements the equivalent of --getbinpkg (-g) without the need for updating the EMERGE_DEFAULT_OPTS variable with the --getbinpkg value:

FILE /etc/portage/make.confEnabling getbinpkg in the FEATURES variable

Verify binary package's OpenGPG signature

This feature is only available in GPKG format.

Portage will try to verify the binary package's signature whenever possible, but users must first set up trusted local keys. A GPG key management tool for portage is in the plan, but for now, users have to manage it manually.

The GPG database of portage is stored in /etc/portage/gnupg by default.

The following script will create a local trusted master key, import key 0x1234567890ABCDEF from, and trust it. Please modify it to a key trusted by the user.

FILE create-portage-local-gpgCreate Portage local trusted master key
PASS="$(openssl rand -base64 32)"
chmod 600 "${KEY_CONFIG_FILE}"
cat > "${KEY_CONFIG_FILE}" <<EOF
     %echo Generating Portage local OpenPGP trust key
     Key-Type: default
     Subkey-Type: default
     Name-Real: Portage Local Trust Key
     Name-Comment: local signing only
     Name-Email: portage@localhost
     Expire-Date: 0
     Passphrase: ${PASS}
     %echo done
mkdir -p "${GNUPGHOME}"
gpg --batch --generate-key "${KEY_CONFIG_FILE}"
rm -f "${KEY_CONFIG_FILE}"
touch "${GNUPGHOME}/pass"
chmod 600 "${GNUPGHOME}/pass"
echo "${PASS}" > "${GNUPGHOME}/pass"
gpg --keyserver "${KEYSERVER}" --recv-keys "${KEY}"
gpg --batch --yes --pinentry-mode loopback --passphrase "${PASS}" --sign-key "${KEY}" 
echo -e "5\ny\n" | gpg --command-fd 0 --edit-key "${KEY}" trust
chmod ugo+r "${GNUPGHOME}/trustdb.gpg"

Make sure /etc/portage/gnupg/trustdb.gpg is global readable, portage will drop root privileges when verifying binary packages.

By default, Portage will only verify GPG signatures when a signature file is found in a package, which allows the user to mix signed and unsigned GPKG binary packages from different sources, and allows to use of old XPAK format binary packages.

If the user needs to force signature verification, the binpkg-request-signature feature needs to be enabled. This feature assumes that all packages should be signed and rejects any unsigned package. Note that this feature does not support per binhost configuration.

FILE /etc/portage/make.confEnabling Portage's binpkg-request-signature feature

Pulling packages from a binary package host

When using a binary package host, clients need to have the PORTAGE_BINHOST variable set in /etc/portage/make.conf or the sync-uri variable in /etc/portage/binrepos.conf. The latter option is preferred over setting the PORTAGE_BINHOST. Without this configuration, the client will not know where the binary packages are stored which results in Portage being unable to retrieve them.

FILE /etc/portage/make.confSetting PORTAGE_BINHOST

The PORTAGE_BINHOST variable uses a space-separated list of URIs. This allows administrators to use several binary package servers simultaneously. The URI must always point to the directory in which the Packages file resides.

FILE /etc/portage/binrepos.confSetting binhost sync-uri
sync-uri =
priority = 10

For each binhost, a name can be configured in the brackets. sync-uri must point to the directory in which the Packages file resides. Optionally, priority can be set. When a package exists in multiple binary package repositories, the package is pulled from the binary package host with the highest priority. This way, a preferred binary package host can be set up.

The support for multiple binary package servers is somewhat incomplete. If several servers serve a binary package for the same package version, then only the first one will be considered. This can be problematic when these binary packages differ in their USE variable configuration and the USE variable configuration of a later binary package would match the systems configuration.

Reinstalling modified binary packages

Passing the --rebuilt-binaries option to emerge will reinstall every binary that has been rebuilt since the package was installed. This is useful in case rebuilding tools like revdep-rebuild are run on the binary package server.

A related option is --rebuilt-binaries-timestamp. It causes emerge not to consider binary packages for a re-install if those binary packages have been built before the given time stamp. This is useful to avoid re-installing all packages, if the binary package server had to be rebuild from scratch but --rebuilt-binaries is used otherwise.

Additional client settings

Next to the getbinpkg feature, Portage also listens to the binpkg-logs feature. This one controls if log files for successful binary package installations should be kept. It is only relevant if the PORT_LOGDIR variable has been set and is enabled by default.

Similar to excluding binary packages for a certain set of packages or categories, clients can be configured to exclude binary package installations for a certain set of packages or categories.

To accomplish this, use the --usepkg-exclude option:

root #emerge -uDNg @world --usepkg-exclude "sys-kernel/gentoo-sources virtual/*"

To enable such additional settings for each emerge command, add the options to the EMERGE_DEFAULT_OPTS variable in the make.conf file:

FILE /etc/portage/make.confEnabling emerge settings on every invocation
EMERGE_DEFAULT_OPTS="${EMERGE_DEFAULT_OPTS} --usepkg-exclude 'sys-kernel/gentoo-sources virtual/*'"

Maintaining binary packages

Exporting and distributing the binary packages will lead to useless storage consumption if the binary package list is not actively maintained.

Removing outdated binary packages

In the app-portage/gentoolkit package an application called eclean is provided. It allows for maintaining Portage-related variable files, such as downloaded source code files, but also binary packages.

The following command will remove all binary packages that have no corresponding ebuild in the installed ebuild repositories:

root #eclean packages

For more details please read the Eclean article.

Another tool that can be used is the qpkg tool from the app-portage/portage-utils package. However, this tool is a bit less configurable.

To clean up unused binary packages (in the sense of used by the server on which the binary packages are stored):

root #qpkg -c

Maintaining the Packages file

Inside the packages directory exists a manifest file called Packages. This file acts as a cache for the metadata of all binary packages in the packages directory. The file is updated whenever Portage adds a binary package to the directory. Similarly, eclean updates it when it removes binary packages.

If for some reason binary packages are simply deleted or copied into the packages directory, or the Packages file gets corrupted or deleted, then it must be recreated. This is done using emaint command:

root #emaint binhost --fix

Advanced topics

Building for other architectures

sys-devel/crossdev is a tool that will easily build cross compile toolchains to allow setting up a binary host for a different architecture, for example for a device like a Raspberry Pi from a more powerful desktop PC.

An installation guide for this package can be found at the Crossdev page.

Build a cross compiler

Using crossdev with the following command can build a toolchain for the desired system:

root #crossdev --stable -t arch-vendor-os-libc

For the rest of this section, the example target will be for a Raspberry Pi 4:

root #crossdev --stable -t aarch64-unknown-linux-gnu

After this has built, a toolchain will have been created in /usr/aarch64-unknown-linux-gnu, and will look like a bare bones Gentoo install where it is possible to edit Portage settings as normal.

Replacing aarch64-unknown-linux-gnu with aarch64-unknown-linux-musl would build a system with the Musl libc rather than Glibc.

Basic setup

Removing the -pam flag from the USE line in /usr/aarch64-unknown-linux-gnu/etc/portage/make.conf is generally recommended in a setup like this:

FILE /usr/aarch64-unknown-linux-gnu/etc/portage/make.confDisable the pam USE flag




CFLAGS="-O2 -pipe -fomit-frame-pointer"

FEATURES="-collision-protect sandbox buildpkg noman noinfo nodoc"
# Ensure pkgs from another repository are not overwritten

#If you want to redefine PORTAGE_TMPDIR uncomment (and/or change the directory location) the following line



List available profiles for the device by running:

root #PORTAGE_CONFIGROOT=/usr/aarch64-unknown-linux-gnu eselect profile list

Next, select the profile that best suits:

root #PORTAGE_CONFIGROOT=/usr/aarch64-unknown-linux-gnu eselect profile set <profile number>

Build a single package

To build a single binary package for use on the device, use the following:

root #emerge-aarch64-linux-unknown-gnu --ask foo

Build world file

To build every package in the world file, then the following command is needed:

root #emerge-aarch64-linux-unknown-gnu --empty-tree @world

Binary location

By default, all binary packages will be stored in /usr/aarch64-unknown-linux-gnu/var/cache/binpkgs, so this is the location needed to be selected when setting up a binary package host.

Creating snapshots of the packages directory

When deploying binary packages for a large number of client systems it might become worthwhile to create snapshots of the packages directory. The client systems then do not use the packages directory directly but use binary packages from the snapshot.

Snapshots can be created using the /usr/lib64/portage/python2.7/binhost-snapshot or /usr/lib64/portage/python3.3/binhost-snapshot tool. It takes four arguments:

  1. A source directory (the path to the packages directory).
  2. A target directory (that must not exist).
  3. A URI.
  4. A binary package server directory.

The files from the package directory are copied to the target directory. A Packages file is then created inside the binary package server directory (fourth argument) with the provided URI.

Client systems need to use an URI that points to the binary package server directory. From there they will be redirected to the URI that was given to binhost-snapshot. This URI has to refer to the target directory.

Understanding the binary package format

XPAK format

XPAK format binary packages created by Portage have the file name ending with .tbz2. These files consist of two parts:

  1. A .tar.bz2 archive containing the files that will be installed on the system.
  2. A xpak archive containing package metadata, the ebuild, and the environment file.

See man xpak for a description of the format.

In app-portage/portage-utils some tools exists that are able to split or create tbz2 and xpak files.

The following command will split the tbz2 into a .tar.bz2 and an .xpak file:

user $qtbz2 -s <package>.tbz2

The .xpak file can be examined using the qxpak utility.

To list the contents:

user $qxpak -l <package>.xpak

The next command will extract a file called USE which contains the enabled USE flags for this package:

user $qxpak -x package-manager-0.xpak USE

GPKG format

GPKG format binary packages created by Portage have the file name ending with .gpkg.tar. These files consist of four parts at least:

  1. A gpkg-1 empty file that used to identify the format.
  2. A C/PV/metadata.tar{.compression} archive containing package metadata, the ebuild, and the environment file.
  3. A C/PV/image.tar{.compression} archive containing the files that will be installed on the system.
  4. A Manifest file containing all files checksum that will be used to check for file corruption.
  5. Multiple optional .sig files containing OpenGPG signature are used for integrity checking and trust management.

The format can be extracted by tar without the need for additional tools.

The PKGDIR layout

The currently used format version 2 has the following layout:

CODE Packages directory layout (version 2)
`+- Packages
 +- app-accessibility/
 |  +- pkg1-version.tbz2
 |  `- pkgN-version.tbz2
 +- app-admin/
 |  `- ...
 `- ...

The Packages file is the major improvement (and also the trigger for Portage to know that the binary package directory uses version 2) over the first binary package directory layout (version 1). In version 1, all binary packages were also hosted inside a single directory (called All/) and the category directories only had symbolic links to the binary packages inside the All/ directory.

In portage-3.0.15 and later, FEATURES=binpkg-multi-instance is enabled by default:

CODE Packages directory layout (version 2 + FEATURES=binpkg-multi-instance)
`+- Packages
 +- app-accessibility/
 |  +- pkg1/
 |    +- pkg1-version-build_id.xpak
 |    `- pkgN-version-build_id.xpak
 +- app-admin/
 |  `- ...
 `- ...

Unpacking with quickunpkg

Zoobab wrote a simple shell tool named quickunpkg to quickly unpack tbz2 files.

External resources

quickpkg man page.