Catalyst

From Gentoo Wiki
Jump to:navigation Jump to:search
Note
This page is about the release building tool. For the AMD graphics drivers, and the configuration tool that comes with them, see fglrx

catalyst is Gentoo's release building tool. With catalyst users are able to completely customize a Gentoo install by customizing the very tools that are used to install a system. Official Gentoo Linux release media is built using Catalyst.

catalyst is capable of:

Installation

USE flags

USE flags for dev-util/catalyst Release metatool used for creating releases based on Gentoo Linux

doc Add extra documentation (API, Javadoc, etc). It is recommended to enable per package instead of globally
iso Pulls in the depends for building iso images

Emerge

To install catalyst, run:

root #emerge --ask dev-util/catalyst

Configuration

Files

After emerging catalyst, the first (and probably only) configuration step is to edit is catalyst.conf.

/etc/catalyst/catalyst.conf
Controls the runtime behavior of catalyst.
/etc/catalyst/catalysstrc
Environment script sourced by catalyst at runtime. It allows users to set options such as HTTP proxies, MAKEOPTS, GENTOO_MIRRORS, or any other environment variables as needed for building.

Usage

Seed tarball

For creating a stage3 a seed tarball is needed. Catalyst will chroot into the seed and emerge the packages for the new stages inside of so that packages build for stage tarballs are isolated from the host system's.

The seed tarball should be placed in the following temporary directory unless the temporary directory location was modified during the Catalyst configuration in the configuration section above. If the temporary directory was modified put the seed in the appropriate directory.

root #mkdir -p /var/tmp/catalyst/builds/default

The seed tarball can be a current stage3 tarball downloaded from one of the Gentoo mirrors.

Relatively recent amd64 tarballs can be found here, and x86 tarballs can be found here.

wget can be used to download from the command line. For example, to download an amd64 tarball:

root #wget https://distfiles.gentoo.org/releases/amd64/autobuilds/current-stage3-amd64/stage3-amd64-YYYYMMDD.tar.xz -O /var/tmp/catalyst/builds/default/stage3-amd64-latest.tar.xz

Snapshot

Local generation

Along with the tarball a snapshot of the Portage tree is required. Catalyst can create a snapshot of the system's current /usr/portage directory with the catalyst -s <date> command.

The snapshot will be saved to /var/tmp/catalyst/snapshots/portage-YYYYMMDD.tar.xz

In the example below 2021.04.17 (April 17, 2021 in an easy to read command-line format) is the argument passed to the --snapshot (-s) option as the snapshot date:

root #catalyst -s 2021.04.17

The <date> argument can be any date format that makes sense to the user. To use the current year/month following format above, simply:

root #catalyst -s $(date +%Y.%m.%d)
Note
Since the file might be created under different name than expected, you may want to link /var/tmp/catalyst/snapshots/gentoo-latest to the snapshot. Ensure the extension (e.g. .tar.xz) match!

Remote download

Another option is to use one of the main repository snapshots available on the mirrors. Current snapshots can be found here.

As mentioned above, the wget command can be used to download a remote snapshot hosted on a HTTP(S) server:

root #wget -O gentoo-latest.tar.xz https://distfiles.gentoo.org/snapshots/gentoo-latest.tar.xz

Specs files

Catalyst uses specs files from which to read parameters to create a stage. Before building a stage3 tarball, stage1 and stage2 tarballs must first be built. Spec file are needed for each of the stages.

The specs files maintained by the Gentoo Release Engineering team can be found in the RelEng git repository. Moving forward, the target of this article is to produce a standard (OpenRC-based) amd64 stage3 tarball using spec files provided by Gentoo's Release Engineering team:

root #wget -O stage1.spec "https://gitweb.gentoo.org/proj/releng.git/plain/releases/specs/amd64/stage1.spec"
root #wget -O stage2.spec "https://gitweb.gentoo.org/proj/releng.git/plain/releases/specs/amd64/stage2.spec"
root #wget -O stage3.spec "https://gitweb.gentoo.org/proj/releng.git/plain/releases/specs/amd64/stage3.spec"

Download the three files above. These specs are very simple; all three of them contain many of the same parameters.

Inside a stage1.spec file

Check out the stage1.spec file, it should look something like this:

FILE stage1.specamd64 stage1 spec file example
subarch: amd64
target: stage1
version_stamp: @TIMESTAMP@
rel_type: default
profile: default/linux/amd64/17.1
snapshot: @TIMESTAMP@
source_subpath: default/stage3-amd64-latest
update_seed: yes
update_seed_command: --update --deep @world
portage_confdir: @REPO_DIR@/releases/portage/stages
portage_prefix: releng

Four lines in the file will need to be modified for the purposes of this guide. After modification it should look like the following:

FILE stage1.specamd64 stage1 spec file example
subarch: amd64
target: stage1
version_stamp: 2021.04.17
rel_type: default
profile: default/linux/amd64/17.1
snapshot: 2021.04.17 
source_subpath: default/stage3-amd64-latest
compression_mode: pixz_x
update_seed: yes
update_seed_command: --update --deep --newuse @world
#portage_confdir: @REPO_DIR@/releases/portage/stages
#portage_prefix: releng

When using the pixz_x compression mode as shown in the above examples, be sure the host has the app-arch/pixz package installed:

root #emerge --ask app-arch/pixz

.spec file option list

The following table provides a list of .spec file options and their descriptions.

Option Description
subarch The subarch can be any of the supported Catalyst sub-architectures. Check the available sub-architectures in /usr/lib/catalyst/arch/${ARCH}.py, ${ARCH} being the arch to be built. subarch simply sets the CHOST and CFLAGS/CXXFLAGS accordingly.
target The target specifies what target Catalyst will build. The example above specifies stage1 as the target to be built. For building a CD, livecd-stage1 should be defied as the target.
version_stamp The version stamp is used as an identifier for the build. On the autobuilds the date is utilized for this setting, however it can be anything the user desires. This parameter will be used on the output tarball, (${target}-${ARCH}-${version_stamp}.tar.bz2), temporary directories, etc...
rel_type rel_type defines what kind of build the system will perform. This is merely another identifier that is available to be used in case (more) differentiation of the builds is needed. If a normal, hardened, and uclibc stages were to be built these could be defined here. Changing it will change the sub-directory inside /var/tmp/catalyst/builds from default to whatever has been set for this value.
profile This is the system profile to be used by Catalyst to build this target. It is specified as a relative path and must be set to one of the system profiles available at /usr/portage/profiles. In the example above the default profile is used (default/linux/amd64/13.0).
snapshot This value specifies which snapshot of main Gentoo repository. See the section above on creating a snapshot for more information on snapshots. If 2015.04 is used like in the example above, catalyst will look for a snapshot available at /var/tmp/catalyst/snapshots/portage-2015.04.tar.bz2.
source_subpath This is the relative path which specifies where the seed stage comes from for this target. In the example above, it will use /var/tmp/catalyst/builds/default/stage3-amd64-latest.tar.bz2 as the seed stage. The prefix of this filesystem path (/var/tmp/catalyst/builds) is determined by the value contained in the storedir variable the /etc/catalyst/catalyst.conf file. The suffix (.tar.bz2) is automatically generated by catalyst at build time.
distcc_hosts These are the hosts used as distcc slaves when distcc is enabled in the /etc/catalyst/catalyst.conf file. It follows the same syntax as distcc-config --set-hosts and is entirely optional. This setting is lacking from the example above.
portage_confdir This is an absolute path where a custom Portage configuration directory can be defined. This comes in handy when desiring to include unstable (~) packages or include a specific make.conf. If this line is left out of the .spec file, then Catalyst will default to using its default configuration. If this line is defined make sure it is defined across all targets (.spec files) to minimize problems.
portage_overlay This option specifies the location to a Portage overlay to use when building the target.
pkgcache_path This allows the optional directory containing the output packages for catalyst. Mainly used as a way for different .spec files to access the same cache directory. Default behavior is for this location to be auto-generated by catalyst based on the .spec file.

On stage2.spec and stage3.spec files, source_subpath needs to reference the stage1 and stage2 tarballs respectively. In other words, with exception of the first .spec file in the sequence, source_subpath should be set to the tarball build previously. For an example, if 2015.04 was used as the version_stamp, the source_subpath for stage2 should be: default/stage1-amd64-2015.04, and the source_subpath for stage3 should be: default/stage2-amd64-2015.04.

Invocation

root #catalyst --help
Catalyst, version 2.0.18
Copyright 2003-2008 Gentoo Foundation
Copyright 2008-2012 various authors
Distributed under the GNU General Public License version 2.1

Usage catalyst [options] [-C variable=value...] [ -s identifier]
 -a --clear-autoresume	clear autoresume flags
 -c --config     use specified configuration file
 -C --cli        catalyst commandline (MUST BE LAST OPTION)
 -d --debug      enable debugging
 -f --file       read specfile
 -F --fetchonly  fetch files only
 -h --help       print this help message
 -p --purge      clear tmp dirs,package cache and autoresume flags
 -P --purgeonly  clear tmp dirs,package cache and autoresume flags and exit
 -T --purgetmponly  clear tmp dirs and autoresume flags and exit
 -s --snapshot   generate a release snapshot
 -V --version    display version information
 -v --verbose    verbose output

Usage examples:

Using the commandline option (-C, --cli) to build a Portage snapshot:
catalyst -C target=snapshot version_stamp=my_date

Using the snapshot option (-s, --snapshot) to build a release snapshot:
catalyst -s 20071121

Using the specfile option (-f, --file) to build a stage target:
catalyst -f stage1-specfile.spec

Building stages

Create distfiles directories

By default, catalyst will download distfiles into the /var/cache/distfiles directory. Make sure it exists before starting the build:

root #mkdir -p /var/cache/distfiles

Start the build

Once the specs have been reviewed, start catalyst by running the following command:

root #catalyst -f stage1.spec && catalyst -f stage2.spec && catalyst -f stage3.spec

If everything went as expected a stage3 should be show up in the /var/tmp/catalyst/builds/default/ directory.

Note
The .spec files may have portage_confdir value set to specify a list of customizations to emerged packages inside the stage files. The directory structure specified by the portage_confdir variable is equivalent to the same layout as the /etc/portage directory. Files inside this directory (i.e. package.use, package.unmask, package.mask, etc.) will need to be manually maintained and occasionally adjusted as gentoo.git continues to be developed. This is the same practice as maintaining a Gentoo installation.

See also

  • Project:RelEng — the official Gentoo project focused on coordinating and improving the creation of official media releases of Gentoo Linux and other Gentoo operating systems.
  • Project:Catalyst/FAQ — contains frequently asked questions (FAQs) relating to the Catalyst tool.
  • Stage tarball — an archive of the basic files used for the installation of Gentoo Linux.