User:Sam/Autoconf cache

From Gentoo Wiki
Jump to:navigation Jump to:search

Background

configure tests can be rather time-intensive, especially on lower-end hardware. It's especially frustrating to see this:

root #./configure
...
checking for size_t... yes
checking for working alloca.h... yes
checking for alloca... yes
checking for dirent.h that defines DIR... yes
checking for library containing opendir... none required
checking for egrep... (cached) /bin/grep -E
checking for sys/wait.h that is POSIX.1 compatible... (cached) yes
checking for an ANSI C-conforming const... yes
checking how to run the C preprocessor... x86_64-pc-linux-gnu-gcc -m32 -E
checking for uid_t in sys/types.h... yes
...

It's possible for configure (autoconf-generated configure, at least) to cache the results of some tests so it doesn't have to re-run them. Note that it's most valuable to cache tests where a small e.g. C program is being compiled and executed as these are often the most expensive.

Implementation within Portage

This was actually implemented within Portage for a period between 2006 and 2008 when it was removed.

It was removed because of various bugs (TODO: links), but the main context is in bug #156308.

There was also a package dev-util/confcache which seems to be on GitHub, but not verified it. I'm curious as to what it did. It ended up being dropped as:

root #git log 62796a056f4480fdbe90ca0d6b0b093a33d1c3a0
commit 62796a056f4480fdbe90ca0d6b0b093a33d1c3a0
Author: Alec Warner <antarus@gentoo.org>
Date:   Mon Oct 30 18:19:35 2006 +0000

    <+ferringb> antarus: just punt the fucking thing, frankly.
[...]

It was hosted at http://gentooexperimental.org/~ferringb/confcache/.

See also bug #433563.

Thoughts

  • Obviously, you can't easily use this across systems and platforms
    • You can kind of do this if you restrict to a subset of tests, but you really want to generate it at least once on each platform, not guess.
  • Need to invalidate cache on major observable changes (which is easy enough to do, just hash a combination of the below, use a symlink and update it when it changes):
    • CHOST
    • Architecture
    • Profile?
    • Toolchain versions changing
  • Exclude more too-specific variables (stuff from libraries which are way too likely to change) (also, it's good enough even with just the basic autoconf ones, and then less likely to fail)
  • Implement it per-package instead to speed up rebuilds and upgrades at least, rather than system-wide
    • ccache analogue here

Setup

Warning
Please don't file bugs when using this configuration for now. Various tests will fail unexpectedly on some packages.

Create a directory for the cache files to reside in:

root #mkdir /etc/portage/confcache

Initialise the cache:

root #touch /etc/portage/confcache/config.cache /etc/portage/confcache/config.cache.x86

Then configure /etc/portage as follows:

FILE /etc/portage/bashrc
# See wiki.gentoo.org/User:Sam/Autoconf_cache
pre_src_configure() {
        if has multilib-minimal ${INHERITED} ; then
                # This is a beauti^W horrible bash intercept for multilib_src_configure
                # We need to define CONFIG_SITE based on the specific (multilib) ABI in use

                # 1. Check if multilib_src_configure is defined before trying to intercept it
                # 2. Make sure we haven't already intercepted it (check for presence of the backup/original: orig_multilib_src_configure)
                if [[ $(type -t multilib_src_configure) == function && $(type -t orig_multilib_src_configure) != function ]] ; then
                        # Backup the original function
                        eval orig_"$(declare -f multilib_src_configure)"

                        # Intercept the multilib-minimal.eclass call to multilib_src_configure
                        multilib_src_configure() {
                                # Needed because e.g. x86 multilib won't match CC
                                if [[ -n ${ABI} && ${ABI} != *amd64* ]] ; then
                                        export CONFIG_SITE="${PORTAGE_CONFIGROOT}/etc/portage/confcache/config.site.${ABI}"
                                else
                                        export CONFIG_SITE="${PORTAGE_CONFIGROOT}/etc/portage/confcache/config.site"
                                fi

                                # Call the original version of the function that we saved
                                orig_multilib_src_configure
                        }
                fi
        else
                export CONFIG_SITE="${PORTAGE_CONFIGROOT}/etc/portage/confcache/config.site"
        fi
}

post_src_configure() {
        unset CONFIG_SITE
}
FILE /etc/portage/confcache/config.site
# (From smaeul's repo)
# Load the cache, but don't tell configure where it is, to keep configure
# from overwriting it.
. "${PORTAGE_CONFIGROOT}/etc/portage/confcache/config.cache"
FILE /etc/portage/confcache/config.site.x86
# (From smaeul's repo)
# Load the cache, but don't tell configure where it is, to keep configure
# from overwriting it.
. "${PORTAGE_CONFIGROOT}/etc/portage/confcache/config.cache.x86"
FILE /etc/sandbox.d/50-confcache
# Allow writing to /etc/portage/confcache, not /etc/portage
# out of caution.
# TODO: I doubt PORTAGE_CONFIGROOT works literally here.
SANDBOX_WRITE="${PORTAGE_CONFIGROOT}/etc/portage/confcache"

Populating the cache

  1. Build a few packages which use autoconf using FEATURES="keepwork"
  2. find /var/tmp/portage/ -iname config.log -exec grep -hrsi "ac_cv" {} \+ | egrep -v "(ac_cv_lib|ac_cv_env|ac_cv_path_install)" | sort -u >> /etc/portage/confcache/config.cache
  3. mv /etc/portage/confcache/config.cache /etc/portage/confcache/config.cache.tmp
  4. sort -u -t= -k1,1 /etc/portage/confcache/config.cache.tmp >> /etc/portage/confcache/config.cache
  5. Manually filter out anything else which looks suspicious from /etc/portage/confcache/config.cache/
  6. Go wild!

Whole system

It may be desirable to run src_configure for everything on the system:

  1. echo "pre_src_compile() { die }" >> /etc/portage/bashrc
  2. FEATURES="keepwork" emerge --jobs=200 -evO @world
  3. sed -i -e '/pre_src_compile/d' /etc/portage/bashrc
  4. find /var/tmp/portage/ -path "*abi_x86_32.x86*" -iname config.log -exec egrep -hrsi "^ac_cv" {} \+ | egrep -v "(ac_cv_lib|ac_cv_env|ac_cv_path_install)" | sort -u >> /etc/portage/confcache/config.cache.x86 ; mv /etc/portage/confcache/config.cache.x86 /etc/portage/confcache/config.cache.x86.tmp ; sort -u -t= -k1,1 /etc/portage/confcache/config.cache.x86.tmp >> /etc/portage/confcache/config.cache.x86
  5. find /var/tmp/portage/ -iname config.log -a -not -path "*abi_x86_32.x86*" -exec egrep -hrsi "^ac_cv" {} \+ | egrep -v "(ac_cv_lib|ac_cv_env|ac_cv_path_install)" | sort -u >> /etc/portage/confcache/config.cache ; mv /etc/portage/confcache/config.cache /etc/portage/confcache/config.cache.tmp ; sort -u -t= -k1,1 /etc/portage/confcache/config.cache.tmp >> /etc/portage/confcache/config.cache

Verification

TODO: explain how to check if it's working

Common issues

Links

Retrieved from "https://wiki.gentoo.org/index.php?title=User:Sam/Autoconf_cache&oldid=1268756"