Catalyst
This page is about the release building tool. For the AMD graphics drivers, and the configuration tool that comes with them, see fglrx
catalyst is Gentoo's release building tool. With catalyst users are able to completely customize a Gentoo install by customizing the very tools that are used to install a system. Official Gentoo Linux release media is built using Catalyst.
catalyst is capable of:
- Building installation stages.
- Building bootable LiveCDs.
- Building netboot images.
Installation
USE flags
USE flags for dev-util/catalyst Release metatool used for creating releases based on Gentoo Linux
Emerge
To install catalyst, run:
root #emerge --ask dev-util/catalystConfiguration
Files
After emerging catalyst, the first (and probably only) configuration step is to edit catalyst.conf.
- /etc/catalyst/catalyst.conf
- Controls the runtime behavior of catalyst.
- /etc/catalyst/catalysstrc
- Environment script sourced by catalyst at runtime. It allows users to set options such as HTTP proxies, MAKEOPTS, GENTOO_MIRRORS, or any other environment variables as needed for building.
Usage
Seed tarball
For creating a stage3 a seed tarball is needed. Catalyst will chroot into the seed and emerge the packages for the new stages inside of so that packages build for stage tarballs are isolated from the host system's.
The seed tarball should be placed in the following temporary directory unless the temporary directory location was modified during the Catalyst configuration in the configuration section above. If the temporary directory was modified put the seed in the appropriate directory.
root #mkdir -p /var/tmp/catalyst/builds/defaultThe seed tarball can be a current stage3 tarball downloaded from one of the Gentoo mirrors.
Relatively recent amd64 OpenRC tarballs can be found here, and their x86 equivalents can be found here.
wget can be used to download from the command line. For example, to download an amd64 tarball:
root #wget https://distfiles.gentoo.org/releases/amd64/autobuilds/current-stage3-amd64/stage3-openrc-amd64-YYYYMMDDTHHMMSSZ.tar.xz -O /var/tmp/catalyst/builds/default/stage3-amd64-openrc-latest.tar.xzSnapshot
Local generation
A snapshot of the ::gentoo ebuild repository (historically called Portage tree) is required. Catalyst can create a snapshot of the system's current /var/db/repo/gentoo directory with the catalyst -s <date> command.
The snapshot will be saved to /var/tmp/catalyst/snapshots/portage-YYYYMMDD.tar.xz
In the example below 2021.04.17 (April 17, 2021 in ISO 8601 compliant format) is the argument passed to the --snapshot (-s) option as the snapshot date:
root #catalyst -s 2021.04.17The <date> argument can be any date format that makes sense to the user. To use the current year/month/day following format above, simply:
root #catalyst -s $(date +%Y.%m.%d)Since the file might be created under different name than expected, it can be helpful to symlink /var/tmp/catalyst/snapshots/gentoo-latest to the snapshot. Ensure the extension (e.g. .tar.xz) match!
Remote download
Another option is to use one of the gentoo:: ebuild repository snapshots available on the mirrors. Current snapshots can be found here.
As mentioned above, the wget command can be used to download a remote snapshot hosted on a HTTP(S) server:
root #wget -O gentoo-latest.tar.xz https://distfiles.gentoo.org/snapshots/gentoo-latest.tar.xzSpecs files
Catalyst uses specs files from which to read parameters to create a stage. Before building a stage3 tarball, stage1 and stage2 tarballs must first be built. Spec file are needed for each of the stages.
The specs files maintained by the Gentoo Release Engineering team can be found in the RelEng git repository. Moving forward, the target of this article is to produce a standard (OpenRC-based) amd64 stage3 tarball using spec files provided by Gentoo's Release Engineering team:
root #wget -O stage1.spec "https://gitweb.gentoo.org/proj/releng.git/plain/releases/specs/amd64/stage1-openrc.spec"
root #wget -O stage3.spec "https://gitweb.gentoo.org/proj/releng.git/plain/releases/specs/amd64/stage3-openrc.spec"
Download the two files above. These specs are very simple; all three of them contain many of the same parameters.
Inside a stage1.spec file
Check out the stage1.spec file, it should look something like this:
stage1.specamd64 stage1 spec file examplesubarch: amd64 target: stage1 version_stamp: openrc-@TIMESTAMP@ rel_type: default profile: default/linux/amd64/17.1 snapshot: @TIMESTAMP@ source_subpath: default/stage3-amd64-openrc-latest compression_mode: pixz update_seed: yes update_seed_command: --update --deep --newuse @world portage_confdir: @REPO_DIR@/releases/portage/stages portage_prefix: releng
Four of the key/value pairs in the file will need to be modified for the purposes of this guide: version_stamp, snapshot, . After modification it should look like the following:
stage1.specamd64 stage1 spec file modifiedsubarch: amd64 target: stage1 version_stamp: 2021.04.17 rel_type: default profile: default/linux/amd64/17.1 snapshot: 2021.04.17 source_subpath: default/stage3-amd64-latest compression_mode: pixz_x update_seed: yes update_seed_command: --update --deep --newuse @world #portage_confdir: @REPO_DIR@/releases/portage/stages #portage_prefix: releng
When using the pixz_x compression mode as shown in the above examples, be sure the host has the app-arch/pixz package installed:
root #emerge --ask app-arch/pixz.spec file option list
The following table provides a list of .spec file options and their descriptions.
| Option | Description |
|---|---|
| subarch | The subarch can be any of the supported Catalyst sub-architectures. Check the available sub-architectures in /usr/lib/catalyst/arch/${ARCH}.py, ${ARCH} being the arch to be built. subarch simply sets the CHOST and CFLAGS/CXXFLAGS accordingly.
|
| target | The target specifies what target Catalyst will build. The example above specifies stage1 as the target to be built. For building a CD, livecd-stage1 should be defied as the target.
|
| version_stamp | The version stamp is used as an identifier for the build. On the autobuilds the date is utilized for this setting, however it can be anything the user desires. This parameter will be used on the output tarball, (${target}-${ARCH}-${version_stamp}.tar.bz2), temporary directories, etc...
|
| rel_type | rel_type defines what kind of build the system will perform. This is merely another identifier that is available to be used in case (more) differentiation of the builds is needed. If a normal, hardened, and uclibc stages were to be built these could be defined here. Changing it will change the sub-directory inside /var/tmp/catalyst/builds from default to whatever has been set for this value.
|
| profile | This is the system profile to be used by Catalyst to build this target. It is specified as a relative path and must be set to one of the system profiles available at /usr/portage/profiles. In the example above the default profile is used (default/linux/amd64/13.0).
|
| snapshot | This value specifies which snapshot of main Gentoo repository. See the section above on creating a snapshot for more information on snapshots. If 2015.04 is used like in the example above, catalyst will look for a snapshot available at /var/tmp/catalyst/snapshots/portage-2015.04.tar.bz2.
|
| source_subpath | This is the relative path which specifies where the seed stage comes from for this target. In the example above, it will use /var/tmp/catalyst/builds/default/stage3-amd64-latest.tar.xz as the seed stage. The prefix of this filesystem path (/var/tmp/catalyst/builds) is determined by the value contained in the storedir variable the /etc/catalyst/catalyst.conf file. The suffix (.tar.bz2) is automatically generated by catalyst at build time.
|
| distcc_hosts | These are the hosts used as distcc slaves when distcc is enabled in the /etc/catalyst/catalyst.conf file. It follows the same syntax as distcc-config --set-hosts and is entirely optional. This setting is lacking from the example above. |
| portage_confdir | This is an absolute path where a custom Portage configuration directory can be defined. This comes in handy when desiring to include unstable (~) packages or include a specific make.conf. If this line is left out of the .spec file, then Catalyst will default to using its default configuration. If this line is defined make sure it is defined across all targets (.spec files) to minimize problems.
|
| portage_overlay | This option specifies the location to a Portage overlay to use when building the target. |
| pkgcache_path | This allows the optional directory containing the output packages for catalyst. Mainly used as a way for different .spec files to access the same cache directory. Default behavior is for this location to be auto-generated by catalyst based on the .spec file. |
On the stage3.spec file, source_subpath value needs to reference the stage1 tarball respectively. In other words, with exception of the first .spec file in the sequence, source_subpath value should be set to the tarball build previously. For an example, if 2021.04.17 was used as the version_stamp value, the source_subpath for the stage1 location should be: default/stage1-amd64-2021.04.17.
Invocation
root #catalyst --helpusage: catalyst [-h] [-V] [-d] [-v] [--log-level {critical,error,warning,notice,info,debug}] [--log-file LOG_FILE] [--color] [--nocolor] [--trace] [--profile] [-a] [-p] [-P] [-T] [--versioned-cachedir]
[--unversioned-cachedir] [-F] [-c CONFIGS] [-f FILE] [-s SNAPSHOT] [-C ...]
options:
-h, --help show this help message and exit
-V, --version display version information
Program output options:
-d, --debug enable debugging (and default --log-level debug)
-v, --verbose verbose output (and default --log-level info)
--log-level {critical,error,warning,notice,info,debug}
set verbosity of output (default: notice)
--log-file LOG_FILE write all output to this file (instead of stdout)
--color colorize output all the time (default: detect)
--nocolor never colorize output all the time (default: detect)
Developer options:
--trace trace program output (akin to `sh -x`)
--profile profile program execution
Temporary file management:
-a, --clear-autoresume
clear autoresume flags
-p, --purge clear tmp dirs, package cache, autoresume flags
-P, --purgeonly clear tmp dirs, package cache, autoresume flags and exit
-T, --purgetmponly clear tmp dirs and autoresume flags and exit
--versioned-cachedir use stage version on cache directory name
--unversioned-cachedir
do not use stage version on cache directory name
Target/config file management:
-F, --fetchonly fetch files only
-c CONFIGS, --configs CONFIGS
use specified configuration files
-f FILE, --file FILE read specfile
-s SNAPSHOT, --snapshot SNAPSHOT
generate a release snapshot
-C ..., --cli ... catalyst commandline (MUST BE LAST OPTION)
Usage examples:
Using the commandline option (-C, --cli) to build a Portage snapshot:
$ catalyst -C target=snapshot version_stamp=my_date
Using the snapshot option (-s, --snapshot) to build a release snapshot:
$ catalyst -s 20071121
Using the specfile option (-f, --file) to build a stage target:
$ catalyst -f stage1-specfile.spec
Building stages
Create distfiles directories
By default, catalyst will download distfiles into the /var/cache/distfiles directory. Make sure it exists before starting the build:
root #mkdir -p /var/cache/distfilesStart the build
Once the specs have been reviewed, start catalyst by running the following command:
root #catalyst -f stage1.spec && catalyst -f stage3.specIf everything went as expected a stage3 should be show up in the /var/tmp/catalyst/builds/default/ directory.
The .spec files may have portage_confdir value set to specify a list of customizations to emerged packages inside the stage files. The directory structure specified by the portage_confdir variable is equivalent to the same layout as the /etc/portage directory. Files inside this directory (i.e. package.use, package.unmask, package.mask, etc.) will need to be manually maintained and occasionally adjusted as gentoo.git continues to be developed. This is the same practice as maintaining a Gentoo installation.
Changes in Catalyst-9999
There are new options to the latest version of dev-util/catalyst which is currently the git version.
Snapshot
Portage snapshots are now handled in git and can be created and updated using:
root #catalyst -s stableYou will see entry that states the Gentoo Portage snapshot name and will look similar to:
catalyst.logOutput exampleNOTICE:catalyst:Creating gentoo tree snapshot 0c5fd9cca1edc63e36234b3dc91c46db24647309 from /var/tmp/catalyst/repos/gentoo.git
In this example take note of 0c5fd9cca1edc63e36234b3dc91c46db24647309 as it will be needed for the spec file.
Spec files
There are some change to the way spec files are now formatted which is shown with stage1 and 3 for a PPC spec file.
stage1-openrc.specppc32subarch: ppc target: stage1 version_stamp: openrc-@TIMESTAMP@ rel_type: default profile: default/linux/ppc/17.0 snapshot_treeish: 0c5fd9cca1edc63e36234b3dc91c46db24647309 source_subpath: default/stage3-ppc-openrc-latest compression_mode: pixz update_seed: yes update_seed_command: --update --deep --newuse @world --jobs 8 --load-average 12 portage_confdir: @REPO_DIR@/releases/portage/stages portage_prefix: releng
stage3-openrc.specppc32subarch: ppc target: stage3 version_stamp: openrc-@TIMESTAMP@ rel_type: default profile: default/linux/ppc/17.0 snapshot_treeish: 0c5fd9cca1edc63e36234b3dc91c46db24647309 source_subpath: default/stage1-ppc-openrc-@TIMESTAMP@ compression_mode: pixz portage_confdir: @REPO_DIR@/releases/portage/stages portage_prefix: releng
Replace 0c5fd9cca1edc63e36234b3dc91c46db24647309 in snapshot_treeish with the commit id that was given when running catalyst -s stable
Jobs and load average
Setting how many job running simultaneously is now set in /etc/catalyst.conf
/etc/catalyst/catalyst.confcatalyst.conf# Integral value passed to emerge as the parameter to --jobs and is used to # define MAKEOPTS during the target build. jobs = 4 # Floating-point value passed to emerge as the parameter to --load-average and # is used to define MAKEOPTS during the target build. # load-average = 4.0
Building stages
Building stages still works the same way, using:
root #catalyst -f stage1.spec && catalyst -f stage3.specTroubleshooting
Error: not a valid profile
The snapshot is likely to have some issue.
- Try generating a snapshot with catalyst -s latest
- Try disabling snapcache in catalyst.conf
See also
- Project:RelEng — the official Gentoo project focused on coordinating and improving the creation of official media releases of Gentoo Linux and other Gentoo operating systems.
- Project:Catalyst/FAQ — contains frequently asked questions (FAQs) relating to the Catalyst tool.
- Stage tarball — an archive of the basic files and packages used for the installation of Gentoo Linux.
- Cataylst Musl Stages Creation - shows how to create your own seed tarballs and edit the spec files to create stage3 using a different libc.
- Catalyst Stage4 Creation