User:Goverp/MasterSlaveGentoo

From Gentoo Wiki
Jump to: navigation, search

Gentoo is a meta-distribution - you configure your own distribution through a combination of profile, USE flag settings, /etc/portage/make.conf settings such as CHOST and packages. Many Gentoo users run the result on just one computer, but it can be easy to use it on several computers using binary packages. This can be done in several ways; this document describes one of them. It may or may not be the most appropriate for any particular situation.

Concepts and Facilities

This document describes one way to use a binary package host. The Binary package guide contains reference details of the Gentoo facilities being used.

One computer, the master does the majority of the "heavy lifting" of compiling the packages. In most cases the others, the slaves, just install the compiled code. portage will compile and install different versions of the code if the master version is unsuitable for a particular slave.

In the approach described, slaves use nfs to access the binary packages on the master. Other access methods such as http are possible: refer to the binary package guide for details.

Matching processor architectures

The master and slaves must have the same CHOST. For best code sharing, their CFLAGS should be similar. For example, the master could be an old but powerful desktop computer with a 4-way AMD Phenom processor, lots of memory and a large RAID disk array, while a slave might be a laptop with a later but smaller AMD a9-9420 processor. It turns out that in most cases code for the Phenom runs well on the a9-9420, and magically, Gentoo can identify the cases where it doesn't and this approach will lead to the slave compiling its own custom code.

It might not work if the master has a later version of the similar architecture; for example, an a9-9420 master and a Phenom slave, unless you compromise by restricting code to the older architecture. This combination is untried; it might work just as well as the other way round, or not. For more details, see #Compiler --march settings for a discussion.

Note
64-bit and 32-bit architectures are dissimilar; 64-bit code won't run on 32-bit processors at all. You could use a 32-bit master for a compatible 64-bit slave, but you'll only be running the slave at half power.

Matching implementation profiles and USE flags

Portage can handle differing profiles and USE flags settings for master and slaves. A slave to use master code for packages with USE settings the same on both machines, while compiling different code where the settings differ.

Different profiles mean many different USE settings between the machines. This may mean the majority of packages have to be recompiled for the slave, negating the benefits of the master/slave approach.

Note
Since Gentoo does not include compiled kernels, this master/slave approach does nothing. The slave will need its own kernel; it may be possible to share a common configuration or compiled code, but that is beyond the scope of this approach.

Installation

Share the portage tree

The master and slave machines may share the same portage tree over NFS. This saves the slave from carrying the portage tree, and saves having to process emerge --sync. However, it's impossible to run any portage commands on the slave when not connected to the master. (This configuration works well when both DISTDIR and PKGDIR have their default values, so the one directory /usr/portage contains everything needed to build code.)

Alternatively, they may share just /usr/portage/packages (or whatever PKGDIR is set to) via NFS, and the each have their own portage tree. In this case, the slave may use emerge --sync against the master's rsyncd - see Local Mirror for details. This approach is a little more cumbersome, but allows the slave to process portage commands whether or not connected to the master.

The following works using the first arrangement under NFSv4, with /home/portage being a bind mount for /usr/portage:

FILE /etc/exports
/home           -ro,no_subtree_check,fsid=root,async,crossmnt   192.168.1.0/24
/home/portage   -ro,no_subtree_check,async,no_root_squash       192.168.1.0/24
FILE /etc/fstab
# <fs>          <mountpoint>    <type>    <opts>         <dump/pass>
 ...
/usr/portage    /home/portage   none      bind,defaults       0 0

(See NFSv4 for more information. NFSv3 needs a slightly different configuration (see NFS).

Share the configuration

/etc/portage contains the configuration. Differing configurations prohibit simply sharing this through NFS. Instead, use rsync, and exploit portage's ability to handle configuration directories instead of files for /etc/portage/make.conf and so on.

Portage processes files in configuration directories in collating sequence order. Give the master files beginning 01... and the slaves files beginning 02.... portage will use the master files as a basis, and override them with the entries in the slave files. Keep the slave's 01 files in step with the master using rsync:

Note
Tailor the hosts definitions and so forth to meet your network configuration and security needs.
FILE /etc/rsyncd.conf
# This line is required by the /etc/init.d/rsyncd script
pid file = /run/rsyncd.pid
use chroot = yes
uid = nobody
gid = nobody
hosts allow = 192.168.1.0/24
hosts deny = 192.168.1.1
read only = yes

[portage-config]
        path = /etc/portage
        comment = Portage configuration
        dont compress = *

# Uncomment the following for your own local portage package tree mirror
#[gentoo-portage]
#        path = /usr/portage
#        comment = Gentoo Portage tree
#        exclude = /distfiles /packages
#        dont compress = *


The following shell script copies all 01-prefixed files of interest. The contents of /etc/portage/env get synchronised 'in toto', as they're not themselves configuration; the relevant part of that is in /etc/portage/package.env. (Substitute master with your master system's network name or address.)

Note
nano backup files get a '~' suffix, so they're excluded in the transfer.
FILE syncportage.shPortage configuration synchronisation shell script
#! /bin/sh

echo Synchronizing /etc/portage

rsync -vr  --exclude '*~' --include 'package.*/' --include='01*' --exclude='*' master::portage-config /etc/portage
rsync -vr  --exclude '*~' --include 'make.conf/' --include='01*' --exclude='*' master::portage-config /etc/portage
rsync -vr  --exclude '*~' master::portage-config/env/ /etc/portage/env

# Uncomment the following line to synchronize the portage package tree; adjust /etc/portage/repos.conf appropriately
# echo Synchronizing portage package tree
# emerge --sync -q

echo Done!

Create the binary package tree

By default /usr/portage/packages, as described in Binary_package_guide#Using_quickpkg.

Use

Master configuration

As described, put all the generic configuration information in files with names prefixed by 01. Here's a couple of examples:

Important
FEATURES includes "buildkg" to save the compiled packages in /usr/portage/packages. Some large packages are excluded by --buildpkg-exclude; note the quotes.
FILE /etc/portage/make.conf/01master
# Make.conf for Acer desktop machine - AMD Phenom 4-way
USE="3dnow 3dnowext cleartype cmake corefonts declarative elogind 
     ...
     sndfile sql sse sse2 sse3 sse4a suid threads vlc wallpapers wayland"

LINGUAS="en_GB en en_US"
L10N="en-GB en en-US"
VIDEO_CARDS="radeon r600"
ABI_X86="64"

# From cpuinfo2cpuflags
CPU_FLAGS_X86="3dnow 3dnowext mmx mmxext popcnt sse sse2 sse3 sse4a"

CHOST="x86_64-pc-linux-gnu"

# Normal mode compilations
CFLAGS="-march=native -O2 -pipe"
CXXFLAGS="${CFLAGS}"

FEATURES="parallel-fetch userfetch userpriv nostrip buildpkg"
EMERGE_DEFAULT_OPTS="--jobs 4 --load-average 4 --quiet-build y --buildpkg-exclude 'sys-kernel/*-sources games-simulation/flightgear-data'"

The master configuration specifies hardware optimizations using CPU_FLAGS_X86 and legacy USE flags. These will be overridden in the slave's configuration.

FILE /etc/portage/package.use/01master
app-office/libreoffice -gtk -gstreamer
app-office/libreoffice-l10n offlinehelp
# Loads of other stuff suppressed

Slave configuration

Files with 02- prefixed names contain the differences between the slave and master configurations.

The majority of the magic lies in /etc/portage/make.conf. Portage executes the slave's 02... configuration after the master's 01..., so its USE above starts with ${USE} to inherit values from master before overriding them. Other settings are straight overrides.

Override DISTDIR to avoid writing to the read-only NFS shared portage tree.

Alter CPU_FLAGS_X86 and USE overriding the legacy settings to allow for differences in master and slave CPUs. For example, Phenom processors support 3dNow!, whereas a9-9420 uses newer speed-ups.

Example:

Important
Do not override CHOST in the slave.
FILE /etc/portage/make.conf/02slave
# Changes (mostly) and additions (use) to 01acer.make

USE="${USE} bluetooth pm-utils upower wireless -3dnow -3dnowext -consolekit elogind"

VIDEO_CARDS="amdgpu radeonsi"
INPUT_DEVICES="synaptics evdev"

CFLAGS="-march=native -O2 -pipe"
CXXFLAGS="${CFLAGS}"

CPU_FLAGS_X86="aes avx avx2 f16c fma3 fma4 mmx mmxext pclmul popcnt sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 xop"

FEATURES="parallel-fetch userfetch nostrip binpkg-logs"

MAKEOPTS="-j2 -l2"

EMERGE_DEFAULT_OPTS="--jobs 2 --load-average 2 --quiet-build y"

# Use a local DISTDIR for stuff not found in networked /usr/portage/distfiles
DISTDIR="/var/tmp/distfiles"
PORTAGE_RO_DISTDIRS="/usr/portage/distfiles"

Other configuration files such as package.use can override and add values as necessary. For example:

FILE /etc/portage/package.use/02slave
# for march=native compilation
sys-kernel/gentoo-sources experimental
# removes dependency on network manager and its ilk
kde-plasma/powerdevil -wireless
# Delete bluetooth support for libreoffice to allow use of the master's code.
app-office/libreoffice -bluetooth
dev-python/PyQt5 -bluetooth
# needed by mesa
x11-libs/libdrm video_cards_radeon

Operation

Run emerge --update .... as normal on the master.

Before running it on the slave, start rsyncd on the master, and run syncportage.sh on the slave. Then start nfs on the master. and on the slave, run:

root #mount /usr/portage
root #emerge --usepkg --ask --update --deep --changed-use --with-bdeps y @world

...output to follow...

Note the purple output, which is portage's way of saying "Here's one I prepared earlier". Notice also the green output where portage calculated that the master package's code is unsuitable for this slave.

Compiler --march settings

The configurations above use --march=native on both master and slave. This works for me. Gcc is at liberty to generate code exploiting hardware features on the master that might not exist on the slave. As it works for me, it appears that it only generates feature-sensitive code if the appropriate value is present in CPU_FLAGS_X86, and portage detects packages sensitive to these flags and compiles separately for the slave. Truly magic. I expected this to end in tears.

If your experience differs, the solution is to choose an explicit, less-specific, value of --march such as k8 for AMD processors.

The code generated, in either case, will get --mtune defaulting to the same architecture, unless overridden. Thus the slave will be running code tuned for the master, and there may be a performance impact, though probably less than for running generic IA-64 or X86 code. You could of course set --mtune to something more generic.