Catalyst

From Gentoo Wiki
Jump to:navigation Jump to:search
Note
This page is about the release building tool. For the AMD graphics drivers, and the configuration tool that comes with them, see fglrx

catalyst is Gentoo's release building tool. With catalyst users are able to completely customize a Gentoo install by customizing the very tools that are used to install a system. Official Gentoo Linux release media is built using Catalyst.

catalyst is capable of:

Installation

USE flags

USE flags for dev-util/catalyst Release metatool used for creating releases based on Gentoo Linux

doc Add extra documentation (API, Javadoc, etc). It is recommended to enable per package instead of globally
iso Pulls in the depends for building iso images

Emerge

To install catalyst, run:

root #emerge --ask dev-util/catalyst

Configuration

Files

After emerging catalyst, the first (and probably only) configuration step is to edit catalyst.conf.

/etc/catalyst/catalyst.conf
Controls the runtime behavior of catalyst.
/etc/catalyst/catalysstrc
Environment script sourced by catalyst at runtime. It allows users to set options such as HTTP proxies, MAKEOPTS, GENTOO_MIRRORS, or any other environment variables as needed for building.

Usage

Seed tarball

For creating a stage3 a seed tarball is needed. Catalyst will chroot into the seed and emerge the packages for the new stages inside of so that packages build for stage tarballs are isolated from the host system's.

The seed tarball should be placed in the following temporary directory unless the temporary directory location was modified during the Catalyst configuration in the configuration section above. If the temporary directory was modified put the seed in the appropriate directory.

root #mkdir -p /var/tmp/catalyst/builds/default

The seed tarball can be a current stage3 tarball downloaded from one of the Gentoo mirrors.

Relatively recent amd64 OpenRC tarballs can be found here, and their x86 equivalents can be found here.

wget can be used to download from the command line. For example, to download an amd64 tarball:

root #wget https://distfiles.gentoo.org/releases/amd64/autobuilds/current-stage3-amd64/stage3-openrc-amd64-YYYYMMDDTHHMMSSZ.tar.xz -O /var/tmp/catalyst/builds/default/stage3-amd64-openrc-latest.tar.xz

Snapshot

Local generation

A snapshot of the ::gentoo ebuild repository (historically called Portage tree) is required. Catalyst can create a snapshot of the system's current /var/db/repo/gentoo directory with the catalyst -s <date> command.

The snapshot will be saved to /var/tmp/catalyst/snapshots/portage-YYYYMMDD.tar.xz

In the example below 2021.04.17 (April 17, 2021 in ISO 8601 compliant format) is the argument passed to the --snapshot (-s) option as the snapshot date:

root #catalyst -s 2021.04.17

The <date> argument can be any date format that makes sense to the user. To use the current year/month/day following format above, simply:

root #catalyst -s $(date +%Y.%m.%d)
Note
Since the file might be created under different name than expected, it can be helpful to symlink /var/tmp/catalyst/snapshots/gentoo-latest to the snapshot. Ensure the extension (e.g. .tar.xz) match!

Remote download

Another option is to use one of the gentoo:: ebuild repository snapshots available on the mirrors. Current snapshots can be found here.

As mentioned above, the wget command can be used to download a remote snapshot hosted on a HTTP(S) server:

root #wget -O gentoo-latest.tar.xz https://distfiles.gentoo.org/snapshots/gentoo-latest.tar.xz

Specs files

Catalyst uses specs files from which to read parameters to create a stage. Before building a stage3 tarball, stage1 and stage2 tarballs must first be built. Spec file are needed for each of the stages.

The specs files maintained by the Gentoo Release Engineering team can be found in the RelEng git repository. Moving forward, the target of this article is to produce a standard (OpenRC-based) amd64 stage3 tarball using spec files provided by Gentoo's Release Engineering team:

root #wget -O stage1.spec "https://gitweb.gentoo.org/proj/releng.git/plain/releases/specs/amd64/stage1-openrc.spec"
root #wget -O stage3.spec "https://gitweb.gentoo.org/proj/releng.git/plain/releases/specs/amd64/stage3-openrc.spec"

Download the two files above. These specs are very simple; all three of them contain many of the same parameters.

Inside a stage1.spec file

Check out the stage1.spec file, it should look something like this:

FILE stage1.specamd64 stage1 spec file example
subarch: amd64
target: stage1
version_stamp: openrc-@TIMESTAMP@
rel_type: default
profile: default/linux/amd64/17.1
snapshot: @TIMESTAMP@
source_subpath: default/stage3-amd64-openrc-latest
compression_mode: pixz
update_seed: yes
update_seed_command: --update --deep --newuse @world
portage_confdir: @REPO_DIR@/releases/portage/stages
portage_prefix: releng

Four of the key/value pairs in the file will need to be modified for the purposes of this guide: version_stamp, snapshot, . After modification it should look like the following:

FILE stage1.specamd64 stage1 spec file modified
subarch: amd64
target: stage1
version_stamp: 2021.04.17
rel_type: default
profile: default/linux/amd64/17.1
snapshot: 2021.04.17
source_subpath: default/stage3-amd64-latest
compression_mode: pixz_x
update_seed: yes
update_seed_command: --update --deep --newuse @world
#portage_confdir: @REPO_DIR@/releases/portage/stages
#portage_prefix: releng

When using the pixz_x compression mode as shown in the above examples, be sure the host has the app-arch/pixz package installed:

root #emerge --ask app-arch/pixz

.spec file option list

The following table provides a list of .spec file options and their descriptions.

Option Description
subarch The subarch can be any of the supported Catalyst sub-architectures. Check the available sub-architectures in /usr/lib/catalyst/arch/${ARCH}.py, ${ARCH} being the arch to be built. subarch simply sets the CHOST and CFLAGS/CXXFLAGS accordingly.
target The target specifies what target Catalyst will build. The example above specifies stage1 as the target to be built. For building a CD, livecd-stage1 should be defied as the target.
version_stamp The version stamp is used as an identifier for the build. On the autobuilds the date is utilized for this setting, however it can be anything the user desires. This parameter will be used on the output tarball, (${target}-${ARCH}-${version_stamp}.tar.bz2), temporary directories, etc...
rel_type rel_type defines what kind of build the system will perform. This is merely another identifier that is available to be used in case (more) differentiation of the builds is needed. If a normal, hardened, and uclibc stages were to be built these could be defined here. Changing it will change the sub-directory inside /var/tmp/catalyst/builds from default to whatever has been set for this value.
profile This is the system profile to be used by Catalyst to build this target. It is specified as a relative path and must be set to one of the system profiles available at /usr/portage/profiles. In the example above the default profile is used (default/linux/amd64/13.0).
snapshot This value specifies which snapshot of main Gentoo repository. See the section above on creating a snapshot for more information on snapshots. If 2015.04 is used like in the example above, catalyst will look for a snapshot available at /var/tmp/catalyst/snapshots/portage-2015.04.tar.bz2.
source_subpath This is the relative path which specifies where the seed stage comes from for this target. In the example above, it will use /var/tmp/catalyst/builds/default/stage3-amd64-latest.tar.xz as the seed stage. The prefix of this filesystem path (/var/tmp/catalyst/builds) is determined by the value contained in the storedir variable the /etc/catalyst/catalyst.conf file. The suffix (.tar.bz2) is automatically generated by catalyst at build time.
distcc_hosts These are the hosts used as distcc slaves when distcc is enabled in the /etc/catalyst/catalyst.conf file. It follows the same syntax as distcc-config --set-hosts and is entirely optional. This setting is lacking from the example above.
portage_confdir This is an absolute path where a custom Portage configuration directory can be defined. This comes in handy when desiring to include unstable (~) packages or include a specific make.conf. If this line is left out of the .spec file, then Catalyst will default to using its default configuration. If this line is defined make sure it is defined across all targets (.spec files) to minimize problems.
portage_overlay This option specifies the location to a Portage overlay to use when building the target.
pkgcache_path This allows the optional directory containing the output packages for catalyst. Mainly used as a way for different .spec files to access the same cache directory. Default behavior is for this location to be auto-generated by catalyst based on the .spec file.

On the stage3.spec file, source_subpath value needs to reference the stage1 tarball respectively. In other words, with exception of the first .spec file in the sequence, source_subpath value should be set to the tarball build previously. For an example, if 2021.04.17 was used as the version_stamp value, the source_subpath for the stage1 location should be: default/stage1-amd64-2021.04.17.

Invocation

root #catalyst --help
usage: catalyst [-h] [-V] [-d] [-v] [--log-level {critical,error,warning,notice,info,debug}] [--log-file LOG_FILE] [--color] [--nocolor] [--trace] [--profile] [-a] [-p] [-P] [-T] [--versioned-cachedir]            
                [--unversioned-cachedir] [-F] [-c CONFIGS] [-f FILE] [-s SNAPSHOT] [-C ...]                                                                                                                          
                                                     
options:                                                                                                  
  -h, --help            show this help message and exit              
  -V, --version         display version information                                                       
                                                                                                          
Program output options:                                                                                   
  -d, --debug           enable debugging (and default --log-level debug)
  -v, --verbose         verbose output (and default --log-level info)       
  --log-level {critical,error,warning,notice,info,debug}
                        set verbosity of output (default: notice)
  --log-file LOG_FILE   write all output to this file (instead of stdout)
  --color               colorize output all the time (default: detect)
  --nocolor             never colorize output all the time (default: detect)
                                                     
Developer options:      
  --trace               trace program output (akin to `sh -x`)
  --profile             profile program execution                                                         
                                                                                                          
Temporary file management:                                                                                
  -a, --clear-autoresume                                                                                  
                        clear autoresume flags
  -p, --purge           clear tmp dirs, package cache, autoresume flags 
  -P, --purgeonly       clear tmp dirs, package cache, autoresume flags and exit
  -T, --purgetmponly    clear tmp dirs and autoresume flags and exit
  --versioned-cachedir  use stage version on cache directory name
  --unversioned-cachedir       
                        do not use stage version on cache directory name
                                                     
Target/config file management:    
  -F, --fetchonly       fetch files only           
  -c CONFIGS, --configs CONFIGS                                                                           
                        use specified configuration files
  -f FILE, --file FILE  read specfile
  -s SNAPSHOT, --snapshot SNAPSHOT
                        generate a release snapshot                                                       
  -C ..., --cli ...     catalyst commandline (MUST BE LAST OPTION)

Usage examples:                                                                                           
                                                     
Using the commandline option (-C, --cli) to build a Portage snapshot:
$ catalyst -C target=snapshot version_stamp=my_date                                                       
                                                     
Using the snapshot option (-s, --snapshot) to build a release snapshot:
$ catalyst -s 20071121

Using the specfile option (-f, --file) to build a stage target:
$ catalyst -f stage1-specfile.spec

Building stages

Create distfiles directories

By default, catalyst will download distfiles into the /var/cache/distfiles directory. Make sure it exists before starting the build:

root #mkdir -p /var/cache/distfiles

Start the build

Once the specs have been reviewed, start catalyst by running the following command:

root #catalyst -f stage1.spec && catalyst -f stage3.spec

If everything went as expected a stage3 should be show up in the /var/tmp/catalyst/builds/default/ directory.

Note
The .spec files may have portage_confdir value set to specify a list of customizations to emerged packages inside the stage files. The directory structure specified by the portage_confdir variable is equivalent to the same layout as the /etc/portage directory. Files inside this directory (i.e. package.use, package.unmask, package.mask, etc.) will need to be manually maintained and occasionally adjusted as gentoo.git continues to be developed. This is the same practice as maintaining a Gentoo installation.

Changes in Catalyst-9999

There are new options to the latest version of dev-util/catalyst which is currently the git version.

Snapshot

Portage snapshots are now handled in git and can be created and updated using:

root #catalyst -s stable

You will see entry that states the Gentoo Portage snapshot name and will look similar to:

FILE catalyst.logOutput example
NOTICE:catalyst:Creating gentoo tree snapshot 0c5fd9cca1edc63e36234b3dc91c46db24647309 from /var/tmp/catalyst/repos/gentoo.git

In this example take note of 0c5fd9cca1edc63e36234b3dc91c46db24647309 as it will be needed for the spec file.

Spec files

There are some change to the way spec files are now formatted which is shown with stage1 and 3 for a PPC spec file.

FILE stage1-openrc.specppc32
subarch: ppc
target: stage1
version_stamp: openrc-@TIMESTAMP@
rel_type: default
profile: default/linux/ppc/17.0
snapshot_treeish: 0c5fd9cca1edc63e36234b3dc91c46db24647309
source_subpath: default/stage3-ppc-openrc-latest
compression_mode: pixz
update_seed: yes
update_seed_command: --update --deep --newuse @world --jobs 8 --load-average 12
portage_confdir: @REPO_DIR@/releases/portage/stages
portage_prefix: releng
FILE stage3-openrc.specppc32
subarch: ppc
target: stage3
version_stamp: openrc-@TIMESTAMP@
rel_type: default
profile: default/linux/ppc/17.0
snapshot_treeish: 0c5fd9cca1edc63e36234b3dc91c46db24647309
source_subpath: default/stage1-ppc-openrc-@TIMESTAMP@
compression_mode: pixz
portage_confdir: @REPO_DIR@/releases/portage/stages
portage_prefix: releng

Replace 0c5fd9cca1edc63e36234b3dc91c46db24647309 in snapshot_treeish with the commit id that was given when running catalyst -s stable

Jobs and load average

Setting how many job running simultaneously is now set in /etc/catalyst.conf

FILE /etc/catalyst/catalyst.confcatalyst.conf
# Integral value passed to emerge as the parameter to --jobs and is used to
# define MAKEOPTS during the target build.
jobs = 4

# Floating-point value passed to emerge as the parameter to --load-average and
# is used to define MAKEOPTS during the target build.
# load-average = 4.0

Building stages

Building stages still works the same way, using:

root #catalyst -f stage1.spec && catalyst -f stage3.spec

Troubleshooting

Error: not a valid profile

The snapshot is likely to have some issue.

  • Try generating a snapshot with catalyst -s latest
  • Try disabling snapcache in catalyst.conf

See also

  • Project:RelEng — the official Gentoo project focused on coordinating and improving the creation of official media releases of Gentoo Linux and other Gentoo operating systems.
  • Project:Catalyst/FAQ — contains frequently asked questions (FAQs) relating to the Catalyst tool.
  • Stage tarball — an archive of the basic files and packages used for the installation of Gentoo Linux.
  • Cataylst Musl Stages Creation - shows how to create your own seed tarballs and edit the spec files to create stage3 using a different libc.
  • Catalyst Stage4 Creation