Catalyst

From Gentoo Wiki
Jump to:navigation Jump to:search
See also
Users looking for the old ATi graphics driver, please see the Fglrx page.

catalyst is Gentoo's release stages and live images building tool. Using catalyst users are able to completely transform their Gentoo installs by customizing the very tools that are used to build the system. Official Gentoo Linux releases are built using catalyst.

catalyst is capable of:

  • Building installation stages
  • Building bootable live images
  • Building netboot images
  • Cross compiling for any of the architectures supported by Gentoo Linux


catalyst is a very powerful tool for advanced usage needs and it can save users a lot of time setting up their systems. For example, user could create a stage3 tarball using the testing branch of Gentoo Linux with any added optimizations they might require, such as LTO, turned on by default. Another very likely case is creating a stage3 tarball for an exotic system setup that Gentoo Linux supports, but, which is too niche to be worth releasing officially supported images. A good example of such a case would be creating a Musl libc stage3 file for the m68k architecture which has a very low number of users even on the standard Gentoo Linux version.


Installation

USE flags

USE flags for dev-util/catalyst Release metatool used for creating releases based on Gentoo Linux

doc Add extra documentation (API, Javadoc, etc). It is recommended to enable per package instead of globally
iso Pulls in the depends for building iso images

Emerge

Warning
Currently Catalyst-3 is no longer supported, as version 4, while still in development, is what RelEng team uses to build all Gentoo stage3 files. The only current option is to use the live ebuild.

To enable the live ebuild during the transition period, add =dev-util/catalyst-9999 atom as below:

FILE /etc/portage/package.accept_keywords/catalystpackage.accept_keyword/catalyst
=dev-util/catalyst-9999 **

To install catalyst run:

root #emerge --ask dev-util/catalyst

Configuration

Files

After emerging catalyst, the first (and probably only) configuration step is to edit catalyst.conf.

/etc/catalyst/catalyst.conf
Controls the runtime behavior of catalyst.
/etc/catalyst/catalystrc
Environment script sourced by catalyst at runtime. It allows users to set options such as HTTP proxies, MAKEOPTS, GENTOO_MIRRORS, or any other environment variables needed for building.

Usage

This section is for general reference to explain how catalyst is used, to learn how to complete particular tasks see Detailed usage examples section.

Seed tarball

For creating a stage3 file a seed tarball is needed. catalyst will chroot into the unpacked seed and emerge the packages needed for the new stages inside of the chroot environment. This way the packages build for new stage tarballs are isolated from the packages on the host system.

The seed tarballs should be placed in the following temporary directory, unless the temporary directory location was modified in the catalyst configuration, see in the configuration section above. If the temporary directory was modified put your seed files in the appropriate directory.

root #mkdir -p /var/tmp/catalyst/builds/default

Any working stage3 tarball can be used as the seed tarball, however it is recommended to use current stage3 tarballs downloaded from one of the official Gentoo Linux mirrors.

Relatively recent amd64 OpenRC tarballs can be found here, and their x86 equivalents can be found here.

wget can be used to download those files from the command line. For example, to download amd64 tarball:

root #wget https://distfiles.gentoo.org/releases/amd64/autobuilds/current-stage3-amd64/stage3-openrc-amd64-YYYYMMDDTHHMMSSZ.tar.xz -O /var/tmp/catalyst/builds/default/stage3-amd64-openrc-latest.tar.xz

Snapshot

Local generation

From version 4 catalyst uses git to clone the current ::gentoo repo and compresses it as a squash file.

The snapshot will be saved to /var/tmp/catalyst/snapshots/gentoo-<COMMIT>.sqfs

Generally, users would only ever want to use the current repo snapshot, so --snapshot stable is all that is needed to be remembered.

root #catalyst --snapshot stable

This will create a squashfile file with the commit reference name like 17e0dbd98931319735970cdb917f7f8b88f647ed, take note of this as it is required when editing spec files.

Older snapshots and manually downloading

Another option is to use one of the gentoo:: squashfile repository snapshots available on the mirrors. Current snapshots can be found here.

For example, to download a remote snapshot hosted on a HTTP(S) server using wget:

Specs files

Catalyst reads all parameters needed to create a stage tarball form spec files. Before building a stage3 tarball, stage1 tarball must be built first. Note that a separate spec file is needed for each of the stages.

The specs files maintained by the Gentoo Release Engineering (RelEng) team can be found in the RelEng git repository. The rest of this section will explain how to edit and repurpose a standard (OpenRC-based) amd64 stage3 tarball using the spec files provided by RelEng team.

Spec files can be downloaded with wget as follows:

root #wget -O stage1.spec "https://gitweb.gentoo.org/proj/releng.git/plain/releases/specs/amd64/stage1-openrc.spec"
root #wget -O stage3.spec "https://gitweb.gentoo.org/proj/releng.git/plain/releases/specs/amd64/stage3-openrc.spec"

Inside a stage1.spec file

Note
@TIMESTAMP@ and @REPODIR@ are settings that aid in automatic stage building using catalyst-auto. For general users it's better to think of these as the system specific changes that need to be made.

Check out the stage1.spec file, it should look something like this:

FILE stage1.specamd64 stage1 spec file example
subarch: amd64
target: stage1
version_stamp: openrc-@TIMESTAMP@
rel_type: default
profile: default/linux/amd64/17.1
snapshot_treeish: @TIMESTAMP@
source_subpath: default/stage3-amd64-openrc-latest
compression_mode: pixz
update_seed: yes
update_seed_command: --update --deep --newuse @world
portage_confdir: /@REPO@DIR/releases/portage/stages
portage_prefix: releng

Generally only four parts need to be changed (version_stamp, snapshot_treeish, source_subpath and portage_confdir) for a successful build. Here is a quick overview of how a modified spec file should look like for a user:

FILE stage1.specamd64 stage1 spec file modified
subarch: amd64
target: stage1
version_stamp: 2023.07.16
rel_type: default
profile: default/linux/amd64/17.1
snapshot_treeish: 17e0dbd98931319735970cdb917f7f8b88f647ed
source_subpath: default/stage3-amd64-latest
compression_mode: pixz_x
update_seed: yes
update_seed_command: --update --deep --newuse @world
portage_confdir: /var/tmp/catalyst/releng/releases/portage/stages
portage_prefix: releng

When using the pixz_x compression mode as shown in the above examples, be sure the host has the app-arch/pixz package installed:

root #emerge --ask app-arch/pixz

More details on what each option does will be shown in the next section.

.spec file option list

The following table provides a list of .spec file options and their descriptions.

Option Description
subarch The subarch simply sets the CHOST and CFLAGS/CXXFLAGS accordingly for the system, finer tuning can be achieved by looking in the toml files located in /usr/share/catalyst/arch.
target The target specifies what target Catalyst will build. The example above specifies stage1 as the target to be built. For building a CD, livecd-stage1 should be defied as the target.
version_stamp The version stamp is used as an identifier for the build. On the autobuilds the date is utilized for this setting, however it can be anything the user desires. This parameter will be used on the output tarball, (${target}-${ARCH}-${version_stamp}.tar.bz2), temporary directories, etc...
rel_type rel_type defines what kind of build the system will perform. This is merely another identifier that is available to be used in case (more) differentiation of the builds is needed. If a normal, hardened, and musl stages were to be built these could be defined here. Changing it will change the sub-directory inside /var/tmp/catalyst/builds from default to whatever has been set for this value.
profile This is the system profile to be used by Catalyst to build this target. It is specified as a relative path and must be set to one of the system profiles available at /usr/portage/profiles. In the example above the default profile is used (default/linux/amd64/17.1).
snapshot_treeish This value specifies which snapshot of main Gentoo repository. See the section above on creating a snapshot for more information on snapshots. If 17e0dbd98931319735970cdb917f7f8b88f647ed is used like in the example above, catalyst will look for a snapshot available at /var/tmp/catalyst/snapshots/gentoo-17e0dbd98931319735970cdb917f7f8b88f647ed.sqfs.
source_subpath This is the relative path which specifies where the seed stage comes from for this target. In the example above, it will use /var/tmp/catalyst/builds/default/stage3-amd64-latest.tar.xz as the seed stage. The prefix of this filesystem path (/var/tmp/catalyst/builds) is determined by the value contained in the storedir variable the /etc/catalyst/catalyst.conf file. The suffix (.tar.bz2) is automatically generated by catalyst at build time.
distcc_hosts These are the hosts used as distcc slaves when distcc is enabled in the /etc/catalyst/catalyst.conf file. It follows the same syntax as distcc-config --set-hosts and is entirely optional. This setting is lacking from the example above and generally recommended not to be used.
portage_confdir This is an absolute path where a custom Portage configuration directory can be defined. This comes in handy when desiring to include unstable (~) packages or include a specific make.conf. If this line is left out of the .spec file, then Catalyst will default to using its default configuration. If this line is defined make sure it is defined across all targets (.spec files) to minimize problems. It is recommended to use the defaults set by RelEng and the edit to the desired needs.
repos This option specifies the location to a Portage overlay to use when building the target.
pkgcache_path This allows the optional directory containing the output packages for catalyst. Mainly used as a way for different .spec files to access the same cache directory. Default behavior is for this location to be auto-generated by catalyst based on the .spec file.
interpreter This is for telling catalyst to use qemu to emulate a different architecture for cross compiling.
portage_prefix This is the directory name set inside of the portage_confdir which will be wiped after the build is finished so the settings needed to building during Catalyst will be removed from the finished product.

On the stage3.spec file, source_subpath value needs to reference the stage1 tarball respectively. In other words, with exception of the first .spec file in the sequence, source_subpath value should be set to the tarball build previously. For an example, if 2021.04.17 was used as the version_stamp value, the source_subpath for the stage1 location should be: default/stage1-amd64-2021.04.17.

Portage config

This is the target's /etc/portage which is almost always needed to be set by the user. It's highly recommended that a user uses the RelEng defaults by cloning them with Git.

user $git clone -o upstream https://github.com/gentoo/releng.git

Where this is stored doesn't matter as long as root can access it, the most common locations are in the user home directory or in /var/tmp/catalyst/releng.

Then point the portage_confdir option to the best defaults for the needed usage, below is listed the different types the RelEng commit hosts and when to use them:

Note
When using the RelEng repo in this way the use of a releng directory is used. This directory is wiped at the end of the build so the end-user's install won't have the same options set on their own machine. If your project needs this to be set then moving them up one level will mean they will be left for the end-user. If this is needed though then creating a custom profile and forcing options on in your project may be the better solution.

Cross Compiling

One of the more useful and advance features is the interpreter option in the spec file. This allows the use of QEMU's user mode to emulate a different architect all together (for example, building for a PowerPC on a x86_64 system).

A requirement to use this is to first setup a working qemu-user system, which can be done using the Embedded Handbook/General/Compiling with qemu user chroot article.

Making use of the stage-qemu defaults in the RelEng's github is also required to set all workarounds needed in portage to allow this to work.

Invocation

root #catalyst --help
                 usage: catalyst [-h] [-V] [--enter-chroot] [-d] [-v]
                [--log-level {critical,error,warning,notice,info,debug}]
                [--log-file LOG_FILE] [--color] [--nocolor] [--trace] [--profile] [-a]
                [-p] [-P] [-T] [--versioned-cachedir] [--unversioned-cachedir] [-F]
                [-c CONFIGS] [-f FILE] [-s SNAPSHOT]

options:
  -h, --help            show this help message and exit
  -V, --version         display version information
  --enter-chroot        Enter chroot before starting the build

Program output options:
  -d, --debug           enable debugging (and default --log-level debug)
  -v, --verbose         verbose output (and default --log-level info)
  --log-level {critical,error,warning,notice,info,debug}
                        set verbosity of output (default: notice)
  --log-file LOG_FILE   write all output to this file (instead of stdout)
  --color               colorize output all the time (default: detect)
  --nocolor             never colorize output all the time (default: detect)

Developer options:
  --trace               trace program output (akin to `sh -x`)
  --profile             profile program execution

Temporary file management:
  -a, --clear-autoresume
                        clear autoresume flags
  -p, --purge           clear tmp dirs, package cache, autoresume flags
  -P, --purgeonly       clear tmp dirs, package cache, autoresume flags and exit
  -T, --purgetmponly    clear tmp dirs and autoresume flags and exit
  --versioned-cachedir  use stage version on cache directory name
  --unversioned-cachedir
                        do not use stage version on cache directory name

Target/config file management:
  -F, --fetchonly       fetch files only
  -c CONFIGS, --configs CONFIGS
                        use specified configuration files
  -f FILE, --file FILE  read specfile
  -s SNAPSHOT, --snapshot SNAPSHOT
                        Make an ebuild repo snapshot

Usage examples:

Using the snapshot option to make a snapshot of the ebuild repo:
$ catalyst --snapshot <git-treeish>

Using the specfile option (-f, --file) to build a stage target:
$ catalyst -f stage1-specfile.spec

Building stages

Distfiles directory

By default, catalyst will download distfiles into the /var/cache/distfiles directory the same way portage does on the host system

Binpkgs directory

Also by default catalyst will create a binary package for each type of build you do (i.e. one for a generic stage3 on amd64 and also one for stage3 tuned for zen3 CPU) on the first build of every new package version, which is very useful to save on build time on the next run. These can be found in /var/tmp/catalyst/packages if using the default location.

For those with space issues then it can be turned off by editing /etc/catalyst/catalyst.confand removing the "pkgcache", line.

Autoresume

If a build fails and it re-run then catalyst will use the existing work directory to continue where it left off from. Sometimes a need arises where a user needs to wipe and start over which is passed with the --clear-autoresume or -a flag. This can also be turned of permanently in /etc/catalyst/catalyst.conf.

Jobs and load average

Setting how many job running simultaneously is set in /etc/catalyst.conf

FILE /etc/catalyst/catalyst.confcatalyst.conf
# Integral value passed to emerge as the parameter to --jobs and is used to
# define MAKEOPTS during the target build.
jobs = 4

# Floating-point value passed to emerge as the parameter to --load-average and
# is used to define MAKEOPTS during the target build.
# load-average = 4.0

Start the build

Once the specs have been reviewed, start catalyst by running the following command:

root #catalyst -f stage1.spec && catalyst -f stage3.spec

If everything went as expected a stage3 should be show up in the /var/tmp/catalyst/builds/default/ directory.

Note
The .spec files may have portage_confdir value set to specify a list of customizations to emerged packages inside the stage files. The directory structure specified by the portage_confdir variable is equivalent to the same layout as the /etc/portage directory. Files inside this directory (i.e. package.use, package.unmask, package.mask, etc.) will need to be manually maintained and occasionally adjusted as gentoo.git continues to be developed. This is the same practice as maintaining a Gentoo installation.

Detailed usage examples

This section contains links to pages that explain how to use catalyst for specific tasks.

Note
This section needs help with examples of creating netboot images or livegui images. Please link them back here and create the new pages under the Catalyst directory.

Standard usage

Catalyst/Stage_Creation - Detailing how to build stage3 and stage4 systems with explanations of how to tune them to the required needs.

Catalyst/Custom_Media_Image - Creating custom bootable images with custom applications for a users needs.

Advanced usage

Catalyst/New_Musl_Stages_Creation - How to create Musl libc stage3 tarballs for unsupported architectures.

See also

  • Project:RelEng — the official Gentoo project focused on coordinating and improving the creation of official media releases of Gentoo Linux and other Gentoo operating systems.
  • Project:Catalyst/FAQ — contains frequently asked questions (FAQs) relating to the Catalyst tool. Note: This should be treated as historic in most cases.
  • Stage tarball — an archive of the basic files and packages used for the installation of Gentoo Linux.