Catalyst
Users looking for the old ATi graphics driver, please see the Fglrx page.
catalyst is Gentoo's release stages and live images building tool. Using catalyst users are able to completely transform their Gentoo installs by customizing the very tools that are used to build the system. Official Gentoo Linux releases are built using catalyst.
catalyst is capable of:
- Building installation stages
- Building bootable live images
- Building netboot images
- Cross compiling for any of the architectures supported by Gentoo Linux
catalyst is a very powerful tool for advanced usage needs and it can save users a lot of time setting up their systems. For example, user could create a stage3 tarball using the testing branch of Gentoo Linux with any added optimizations they might require, such as LTO, turned on by default. Another very likely case is creating a stage3 tarball for an exotic system setup that Gentoo Linux supports, but, which is too niche to be worth releasing officially supported images. A good example of such a case would be creating a Musl libc stage3 file for the m68k architecture which has a very low number of users even on the standard Gentoo Linux version.
Installation
USE flags
USE flags for dev-util/catalyst Release metatool used for creating releases based on Gentoo Linux
Emerge
Currently Catalyst-3 is no longer supported, as version 4, while still in development, is what RelEng team uses to build all Gentoo stage3 files. The only current option is to use the live ebuild.
To enable the live ebuild during the transition period, add =dev-util/catalyst-9999 atom as below:
/etc/portage/package.accept_keywords/catalystpackage.accept_keyword/catalyst=dev-util/catalyst-9999 **
To install catalyst run:
root #emerge --ask dev-util/catalystConfiguration
Files
After emerging catalyst, the first (and probably only) configuration step is to edit catalyst.conf.
- /etc/catalyst/catalyst.conf
- Controls the runtime behavior of catalyst.
- /etc/catalyst/catalystrc
- Environment script sourced by catalyst at runtime. It allows users to set options such as HTTP proxies, MAKEOPTS, GENTOO_MIRRORS, or any other environment variables needed for building.
Usage
This section is for general reference to explain how catalyst is used, to learn how to complete particular tasks see Detailed usage examples section.
Seed tarball
For creating a stage3 file a seed tarball is needed. catalyst will chroot into the unpacked seed and emerge the packages needed for the new stages inside of the chroot environment. This way the packages build for new stage tarballs are isolated from the packages on the host system.
The seed tarballs should be placed in the following temporary directory, unless the temporary directory location was modified in the catalyst configuration, see in the configuration section above. If the temporary directory was modified put your seed files in the appropriate directory.
root #mkdir -p /var/tmp/catalyst/builds/defaultAny working stage3 tarball can be used as the seed tarball, however it is recommended to use current stage3 tarballs downloaded from one of the official Gentoo Linux mirrors.
Relatively recent amd64 OpenRC tarballs can be found here, and their x86 equivalents can be found here.
wget can be used to download those files from the command line. For example, to download amd64 tarball:
root #wget https://distfiles.gentoo.org/releases/amd64/autobuilds/current-stage3-amd64/stage3-openrc-amd64-YYYYMMDDTHHMMSSZ.tar.xz -O /var/tmp/catalyst/builds/default/stage3-amd64-openrc-latest.tar.xzSnapshot
Local generation
From version 4 catalyst uses git to clone the current ::gentoo repo and compresses it as a squash file.
The snapshot will be saved to /var/tmp/catalyst/snapshots/gentoo-<COMMIT>.sqfs
Generally, users would only ever want to use the current repo snapshot, so --snapshot stable is all that is needed to be remembered.
root #catalyst --snapshot stableThis will create a squashfile file with the commit reference name like 17e0dbd98931319735970cdb917f7f8b88f647ed, take note of this as it is required when editing spec files.
Older snapshots and manually downloading
Another option is to use one of the gentoo:: squashfile repository snapshots available on the mirrors. Current snapshots can be found here.
For example, to download a remote snapshot hosted on a HTTP(S) server using wget:
Specs files
Catalyst reads all parameters needed to create a stage tarball form spec files. Before building a stage3 tarball, stage1 tarball must be built first. Note that a separate spec file is needed for each of the stages.
The specs files maintained by the Gentoo Release Engineering (RelEng) team can be found in the RelEng git repository. The rest of this section will explain how to edit and repurpose a standard (OpenRC-based) amd64 stage3 tarball using the spec files provided by RelEng team.
Spec files can be downloaded with wget as follows:
root #wget -O stage1.spec "https://gitweb.gentoo.org/proj/releng.git/plain/releases/specs/amd64/stage1-openrc.spec"
root #wget -O stage3.spec "https://gitweb.gentoo.org/proj/releng.git/plain/releases/specs/amd64/stage3-openrc.spec"
Inside a stage1.spec file
@TIMESTAMP@ and @REPODIR@ are settings that aid in automatic stage building using catalyst-auto. For general users it's better to think of these as the system specific changes that need to be made.
Check out the stage1.spec file, it should look something like this:
stage1.specamd64 stage1 spec file examplesubarch: amd64
target: stage1
version_stamp: openrc-@TIMESTAMP@
rel_type: default
profile: default/linux/amd64/17.1
snapshot_treeish: @TIMESTAMP@
source_subpath: default/stage3-amd64-openrc-latest
compression_mode: pixz
update_seed: yes
update_seed_command: --update --deep --newuse @world
portage_confdir: /@REPO@DIR/releases/portage/stages
portage_prefix: releng
Generally only four parts need to be changed (version_stamp, snapshot_treeish, source_subpath and portage_confdir) for a successful build. Here is a quick overview of how a modified spec file should look like for a user:
stage1.specamd64 stage1 spec file modifiedsubarch: amd64
target: stage1
version_stamp: 2023.07.16
rel_type: default
profile: default/linux/amd64/17.1
snapshot_treeish: 17e0dbd98931319735970cdb917f7f8b88f647ed
source_subpath: default/stage3-amd64-latest
compression_mode: pixz_x
update_seed: yes
update_seed_command: --update --deep --newuse @world
portage_confdir: /var/tmp/catalyst/releng/releases/portage/stages
portage_prefix: releng
When using the pixz_x compression mode as shown in the above examples, be sure the host has the app-arch/pixz package installed:
root #emerge --ask app-arch/pixzMore details on what each option does will be shown in the next section.
.spec file option list
The following table provides a list of .spec file options and their descriptions.
| Option | Description |
|---|---|
| subarch | The subarch simply sets the CHOST and CFLAGS/CXXFLAGS accordingly for the system, finer tuning can be achieved by looking in the toml files located in /usr/share/catalyst/arch.
|
| target | The target specifies what target Catalyst will build. The example above specifies stage1 as the target to be built. For building a CD, livecd-stage1 should be defied as the target.
|
| version_stamp | The version stamp is used as an identifier for the build. On the autobuilds the date is utilized for this setting, however it can be anything the user desires. This parameter will be used on the output tarball, (${target}-${ARCH}-${version_stamp}.tar.bz2), temporary directories, etc...
|
| rel_type | rel_type defines what kind of build the system will perform. This is merely another identifier that is available to be used in case (more) differentiation of the builds is needed. If a normal, hardened, and musl stages were to be built these could be defined here. Changing it will change the sub-directory inside /var/tmp/catalyst/builds from default to whatever has been set for this value.
|
| profile | This is the system profile to be used by Catalyst to build this target. It is specified as a relative path and must be set to one of the system profiles available at /usr/portage/profiles. In the example above the default profile is used (default/linux/amd64/17.1).
|
| snapshot_treeish | This value specifies which snapshot of main Gentoo repository. See the section above on creating a snapshot for more information on snapshots. If 17e0dbd98931319735970cdb917f7f8b88f647ed is used like in the example above, catalyst will look for a snapshot available at /var/tmp/catalyst/snapshots/gentoo-17e0dbd98931319735970cdb917f7f8b88f647ed.sqfs. |
| source_subpath | This is the relative path which specifies where the seed stage comes from for this target. In the example above, it will use /var/tmp/catalyst/builds/default/stage3-amd64-latest.tar.xz as the seed stage. The prefix of this filesystem path (/var/tmp/catalyst/builds) is determined by the value contained in the storedir variable the /etc/catalyst/catalyst.conf file. The suffix (.tar.bz2) is automatically generated by catalyst at build time.
|
| distcc_hosts | These are the hosts used as distcc slaves when distcc is enabled in the /etc/catalyst/catalyst.conf file. It follows the same syntax as distcc-config --set-hosts and is entirely optional. This setting is lacking from the example above and generally recommended not to be used. |
| portage_confdir | This is an absolute path where a custom Portage configuration directory can be defined. This comes in handy when desiring to include unstable (~) packages or include a specific make.conf. If this line is left out of the .spec file, then Catalyst will default to using its default configuration. If this line is defined make sure it is defined across all targets (.spec files) to minimize problems. It is recommended to use the defaults set by RelEng and the edit to the desired needs.
|
| repos | This option specifies the location to a Portage overlay to use when building the target. |
| pkgcache_path | This allows the optional directory containing the output packages for catalyst. Mainly used as a way for different .spec files to access the same cache directory. Default behavior is for this location to be auto-generated by catalyst based on the .spec file. |
| interpreter | This is for telling catalyst to use qemu to emulate a different architecture for cross compiling. |
| portage_prefix | This is the directory name set inside of the portage_confdir which will be wiped after the build is finished so the settings needed to building during Catalyst will be removed from the finished product.
|
On the stage3.spec file, source_subpath value needs to reference the stage1 tarball respectively. In other words, with exception of the first .spec file in the sequence, source_subpath value should be set to the tarball build previously. For an example, if 2021.04.17 was used as the version_stamp value, the source_subpath for the stage1 location should be: default/stage1-amd64-2021.04.17.
Portage config
This is the target's /etc/portage which is almost always needed to be set by the user. It's highly recommended that a user uses the RelEng defaults by cloning them with Git.
user $git clone -o upstream https://github.com/gentoo/releng.gitWhere this is stored doesn't matter as long as root can access it, the most common locations are in the user home directory or in /var/tmp/catalyst/releng.
Then point the portage_confdir option to the best defaults for the needed usage, below is listed the different types the RelEng commit hosts and when to use them:
- releases/portage/iso - For creating livecds based on the minimal and admincd.
- releases/portage/iso-qemu - The same as above but for when compiled in QEMU for a different architecture than the host's.
- releases/portage/livegui - Defaults for creating a working livegui ISO.
- releases/portage/stages - Defaults for creating a custom Stage3 or 4 tarball.
- releases/portage/stages-qemu - The same as above but for when compiled in QEMU for a different architecture than the host's.
When using the RelEng repo in this way the use of a releng directory is used. This directory is wiped at the end of the build so the end-user's install won't have the same options set on their own machine. If your project needs this to be set then moving them up one level will mean they will be left for the end-user. If this is needed though then creating a custom profile and forcing options on in your project may be the better solution.
Cross Compiling
One of the more useful and advance features is the interpreter option in the spec file. This allows the use of QEMU's user mode to emulate a different architect all together (for example, building for a PowerPC on a x86_64 system).
A requirement to use this is to first setup a working qemu-user system, which can be done using the Embedded Handbook/General/Compiling with qemu user chroot article.
Making use of the stage-qemu defaults in the RelEng's github is also required to set all workarounds needed in portage to allow this to work.
Invocation
root #catalyst --help usage: catalyst [-h] [-V] [--enter-chroot] [-d] [-v]
[--log-level {critical,error,warning,notice,info,debug}]
[--log-file LOG_FILE] [--color] [--nocolor] [--trace] [--profile] [-a]
[-p] [-P] [-T] [--versioned-cachedir] [--unversioned-cachedir] [-F]
[-c CONFIGS] [-f FILE] [-s SNAPSHOT]
options:
-h, --help show this help message and exit
-V, --version display version information
--enter-chroot Enter chroot before starting the build
Program output options:
-d, --debug enable debugging (and default --log-level debug)
-v, --verbose verbose output (and default --log-level info)
--log-level {critical,error,warning,notice,info,debug}
set verbosity of output (default: notice)
--log-file LOG_FILE write all output to this file (instead of stdout)
--color colorize output all the time (default: detect)
--nocolor never colorize output all the time (default: detect)
Developer options:
--trace trace program output (akin to `sh -x`)
--profile profile program execution
Temporary file management:
-a, --clear-autoresume
clear autoresume flags
-p, --purge clear tmp dirs, package cache, autoresume flags
-P, --purgeonly clear tmp dirs, package cache, autoresume flags and exit
-T, --purgetmponly clear tmp dirs and autoresume flags and exit
--versioned-cachedir use stage version on cache directory name
--unversioned-cachedir
do not use stage version on cache directory name
Target/config file management:
-F, --fetchonly fetch files only
-c CONFIGS, --configs CONFIGS
use specified configuration files
-f FILE, --file FILE read specfile
-s SNAPSHOT, --snapshot SNAPSHOT
Make an ebuild repo snapshot
Usage examples:
Using the snapshot option to make a snapshot of the ebuild repo:
$ catalyst --snapshot <git-treeish>
Using the specfile option (-f, --file) to build a stage target:
$ catalyst -f stage1-specfile.spec
Building stages
Distfiles directory
By default, catalyst will download distfiles into the /var/cache/distfiles directory the same way portage does on the host system
Binpkgs directory
Also by default catalyst will create a binary package for each type of build you do (i.e. one for a generic stage3 on amd64 and also one for stage3 tuned for zen3 CPU) on the first build of every new package version, which is very useful to save on build time on the next run. These can be found in /var/tmp/catalyst/packages if using the default location.
For those with space issues then it can be turned off by editing /etc/catalyst/catalyst.confand removing the "pkgcache", line.
Autoresume
If a build fails and it re-run then catalyst will use the existing work directory to continue where it left off from. Sometimes a need arises where a user needs to wipe and start over which is passed with the --clear-autoresume or -a flag. This can also be turned of permanently in /etc/catalyst/catalyst.conf.
Jobs and load average
Setting how many job running simultaneously is set in /etc/catalyst.conf
/etc/catalyst/catalyst.confcatalyst.conf# Integral value passed to emerge as the parameter to --jobs and is used to
# define MAKEOPTS during the target build.
jobs = 4
# Floating-point value passed to emerge as the parameter to --load-average and
# is used to define MAKEOPTS during the target build.
# load-average = 4.0
Start the build
Once the specs have been reviewed, start catalyst by running the following command:
root #catalyst -f stage1.spec && catalyst -f stage3.specIf everything went as expected a stage3 should be show up in the /var/tmp/catalyst/builds/default/ directory.
The .spec files may have portage_confdir value set to specify a list of customizations to emerged packages inside the stage files. The directory structure specified by the portage_confdir variable is equivalent to the same layout as the /etc/portage directory. Files inside this directory (i.e. package.use, package.unmask, package.mask, etc.) will need to be manually maintained and occasionally adjusted as gentoo.git continues to be developed. This is the same practice as maintaining a Gentoo installation.
Detailed usage examples
This section contains links to pages that explain how to use catalyst for specific tasks.
This section needs help with examples of creating netboot images or livegui images. Please link them back here and create the new pages under the Catalyst directory.
Standard usage
Catalyst/Stage_Creation - Detailing how to build stage3 and stage4 systems with explanations of how to tune them to the required needs.
Catalyst/Custom_Media_Image - Creating custom bootable images with custom applications for a users needs.
Advanced usage
Catalyst/New_Musl_Stages_Creation - How to create Musl libc stage3 tarballs for unsupported architectures.
See also
- Project:RelEng — the official Gentoo project focused on coordinating and improving the creation of official media releases of Gentoo Linux and other Gentoo operating systems.
- Project:Catalyst/FAQ — contains frequently asked questions (FAQs) relating to the Catalyst tool. Note: This should be treated as historic in most cases.
- Stage tarball — an archive of the basic files and packages used for the installation of Gentoo Linux.