Project:Toolchain/libcrypt implementation

From Gentoo Wiki
Jump to:navigation Jump to:search

glibc provides many interfaces and there's a steady process to deprecate some of these built-in implementations to allow other providers on the system to better fulfill their purpose. Eventually, glibc will remove its libcrypt implementation, so this migration is unavoidable.

Gentoo has already done this before with:

This time, we're working on replacing sys-libs/glibc's libcrypt with sys-libs/libxcrypt. This effort is being tracked in bug #699422.

TL;DR

  • Most people won't have to do anything special at all, just sync and do a world upgrade!
  • Check out Troubleshooting if you hit a problem.
    • The most common issue is needing to use emerge --ignore-built-slot-operator-deps=y -uvDU @world temporarily, then doing world again without that special --ignore-built-slot-operator-deps=y option.
    • The second most common issue is needing to upgrade Python to not incorrectly include crypt.h and link against libcrypt (which may break anything trying to use libpython): emerge -uv1 --ignore-built-slot-operator-deps=y dev-lang/python:3.8 dev-lang/python:3.9 should be sufficient on an already up to date system, but use this only if truly necessary. Recommend running first.

Migrating the system

Preliminaries

Don't skip these!

  1. Ensure FEATURES="preserve-libs" (default) has been set for now (bug #802207)!
  2. If a user's password was set before the year 2008 (bug #802267), the password may need changed in order to force it to be protected using newer mechanisms than e.g. md5crypt. A new sys-libs/pam version has been added to avoid this issue (bug #802807) although issues are being investigated (bug #803050).
  3. Sync your copy of the repository (emerge --sync), especially to avoid issues like bug #802210.
  4. Fully upgrade the system (emerge -a -uvDU @world or similar). This is critical to mitigate possible Portage issues.
  5. Depclean (emerge -acv).

/etc/portage changes

Warning
This is now obsolete since stabilisation on 2021-11-01, see bug #809410.

This section is now obsolete for ~arch users since the changes have now been made in the Gentoo ebuild repository. It is still relevant if you're using stable.

FILE /etc/portage/package.use
# Disable libcrypt in glibc
>=sys-libs/glibc-2.33-r3 -crypt

# Provide libcrypt
>=sys-libs/libxcrypt-4.4.23-r2 system
FILE /etc/portage/package.accept_keywords
# Allow the new libcrypt virtual which includes libxcrypt
~virtual/libcrypt-2

# Needed for the right masks to kick in, we revbumped for the migration
# NOTE: Remember to drop this once the migration is done as a stable user
=sys-libs/glibc-2.33-r7
>=sys-libs/libxcrypt-4.4.24
FILE /etc/portage/package.unmask
# Allow virtual which specifies libxcrypt
~virtual/libcrypt-2
FILE /etc/portage/profile/package.use.mask
# Allow libxcrypt to be the system provider of libcrypt, not glibc
>=sys-libs/libxcrypt-4.4.23-r2 -system -split-usr
FILE /etc/portage/profile/package.use.force
# Don't force glibc to provide libcrypt
>=sys-libs/glibc-2.33-r3 -crypt

Upgrade system

Perform a full world upgrade again. If all is going well, there should be many rebuilds caused by virtual/libcrypt.

root #emerge -p -uvDU @world

Troubleshooting

General tips before getting into specific situations:

  • Use more backtracking if needed (e.g. --backtrack=9999)
  • Disable binpkgs temporarily (--usepkg=n)

error: crypt.h: No such file or directory

It is possible to hit an error such as: 

/usr/include/python3.9/Python.h:44:10: fatal error: crypt.h: No such file or directory
   44 | #include <crypt.h>
      |          ^~~~~~~~~

Such errors are likely to be caused by a Portage scheduling issue - bug #802210. A workaround for this has been introduced into >=sys-libs/glibc-2.33-r5 which temporarily preserves the crypt.h header (removed because of the confusion with collisions).

Newer versions of each Python in Gentoo no longer unnecessarily link against libcrypt or try to include crypt.h:

  1. emerge -v1 dev-lang/python:3.8 dev-lang/python:3.9 (and python:3.10 if you have that installed already)
  2. emerge -v1 libxcrypt

Perl fails when trying to emerge libxcrypt

If you have an newer version of glibc installed already with -crypt, you'll need to employ a workaround in order to trigger the preservation mechanism again:

FILE /etc/portage/package.mask
# Mask newer glibc *temporarily*
>=sys-libs/glibc-2.33-r7
FILE /etc/portage/package.unmask
# Allow virtual which specifies glibc *temporarily*
virtual/libcrypt:0/1
FILE /etc/portage/package.mask
# Forbid libxcrypt virtual *temporarily*
virtual/libcrypt:0/2
root #USE="crypt" emerge -v1O sys-libs/glibc --autounmask=n --usepkg=n

Now delete the aforementioned entries just created. Then:

root #USE="-crypt" emerge -v1O sys-libs/glibc --autounmask=n --usepkg=n

Please only do this if you are unable to merge libxcrypt itself with a Perl error.

Python can't be re-emerged

Re-emerging Python may rarely give an error like:

root #emerge -v1 dev-lang/python
[...]
*** WARNING: renaming "_crypt" since importing it failed: build/lib.linux-x86_64-3.9/_crypt.cpython-39-x86_64-linux-gnu.so: undefined symbol: crypt

In this case, please run: 1. emerge -v1O sys-libs/libxcrypt 2. emerge -v1O dev-lang/python:3.9 only if your existing Python wasn't too old (make sure you already had e.g. Python 3.9). Otherwise, skip straight to step 3. 3. Run a world upgrade/try again with your world upgrade.

Conflicts: Nudge the virtual

In some cases, confusing conflicts have appeared.

If this happens, it is recommended to nudge the virtual first:

root #emerge -v1 "~virtual/libcrypt-2" --autounmask=n --usepkg=n

Then complete a full world upgrade.

Conflicts: static-libs conflicts

Even with the nudge, you may still get some conflicts. It's possible they're to do with static-libs.

root #emerge -v1 "~virtual/libcrypt-2" --autounmask=n --usepkg=n --backtrack=9999

Make sure that static-libs consistently enabled on both sys-libs/libxcrypt and virtual/libcrypt if a package e.g. sys-apps/util-linux or sys-apps/busybox requires it:

FILE /etc/portage/package.use
# Disable libcrypt in glibc
sys-libs/glibc -crypt

# Provide libcrypt
sys-libs/libxcrypt system static-libs
virtual/libcrypt static-libs

Then try the nudge again, then a full world upgrade.

Conflicts: General (try with explicit reverse dependencies)

Where the above --usepkg=n and --backtrack=9999 did not help rebuilding all reverse dependencies of virtual/libcrypt, handing that list to portage explicitly has been proven to work in some cases. All it takes is either qdepends (of app-portage/portage-utils) or equery (from app-portage/gentoolkit) to gather the list and glueing things together in shell:

For the variant using qdepends:

root #emerge -1 '~virtual/libcrypt-2' $(qdepends -Q virtual/libcrypt | awk -F: '{print $1}' | sed 's,^,=,')

For the variant using equery:

root #emerge -1 '~virtual/libcrypt-2' $(equery depends virtual/libcrypt | awk '{print $1}' | sed 's,^,=,')

What this does is collect reverse dependencies, take the first word of each line, prepend "=" to each line, and forward the resulting lines to emerge.

This works because we're forcing Portage to consider rebuilding both libxcrypt and any packages which must be rebuilt together with it simultaneously.

Circular dependencies

Tip
This is a prominent issue when performing a fresh install with a stage3 then upgrading immediately to ~arch.

A Portage (arguably PMS) bug bug #702806 (and its many duplicates) mean that the python-any-r1 eclass pulls in the latest version of Python even if it is not listed in PYTHON_TARGETS.

See Circular_dependencies#Solution_2 for more details / fixes!

This can manifest as:

root #emerge -p -uvDU @world
[...]

* Error: circular dependencies:

(dev-lang/python-3.10.0_beta4:3.10/3.10::gentoo, ebuild scheduled for merge) depends on
 (virtual/libcrypt-2:0/2::gentoo, ebuild scheduled for merge) (buildtime_slot_op)
  (sys-libs/libxcrypt-4.4.23-r1:0/1::gentoo, ebuild scheduled for merge) (runtime)
   (sys-libs/glibc-2.33-r2:2.2/2.2::gentoo, ebuild scheduled for merge) (buildtime)
    (dev-lang/python-3.10.0_beta4:3.10/3.10::gentoo, ebuild scheduled for merge) (buildtime)

In this case, please try emerge -a -uvDU @world --exclude python:3.10.

Collisions

  • Collisions on just libcrypt.h where libcrypt.h is not owned by any package was possible for a time although the workaround has since been removed. We recommend using FEATURES="unmerge-orphans" and not FEATURES="collision-protect".
  • Collisions between glibc and libxcrypt are the result of a possible Portage bug. Work around this with FEATURES="-collision-protect -unmerge-orphans -protect-owned" emerge -v1 libxcrypt.

Blockers

On one occasion, a nonsensical blocker occurred (on a more outdated system):

root #emerge -p -uvDU @world --backtrack=9999

Do NOT attempt to unmerge sys-libs/glibc.

In this case, emerge -av1 virtual/libcrypt sys-libs/libxcrypt was sufficient to install the new sys-libs/glibc[-crypt] and allow a subsequent world upgrade to be cleanly calculated (even if it looks scary):

root #emerge -av1 virtual/libcrypt sys-libs/libxcrypt
Masking previous subslot
Note
Please remember to mention this if seeking support and remember to delete the entry if it doesn't seem to help, to avoid confusion.

In some cases, it's helped to mask the older subslot of virtual/libcrypt when faced with conflicts that cannot be solved by the other methods described here. A high --backtrack value is likely to be needed still!

FILE /etc/portage/package.mask
# Attempt to avoid conflicts for libxcrypt migration
virtual/libcrypt:0/1
Last resort
Warning
Only do this if all other options have been attempted. This is a last-resort option after troubleshooting in e.g. forums and support channels has failed.

If the system is in a state where part of it has been rebuilt against libxcrypt and part has not, emerge output will appear similar to:

root #emerge -p -uvDU @world

Note that in the previous output, some packages expected :0/1, while some expected :0/2. The initial investigation should be focused on trying to emerge --oneshot --verbose packages stuck on :0/1 to investigate why emerge cannot consider for them for upgrades.

If such investigation fails, it may be necessary to use a sledgehammer approach. Avoid this unless completely necessary:

root #emerge -p -uvDU @world --ignore-built-slot-operator-deps=y

Then proceed to attempt a world upgrade without the --ignore-built-slot-operator-deps=y option. If it fails, please refer to Conflicts: try with explicit reverse dependencies.

Migrating tree-wide

Expected problems

  • Largely should be smooth when building applications
    • Fedora switched in 2018 and have hit few issues, upstreaming patches where they did (rarely) hit something
  • Missing dependencies on virtual/libcrypt (or missing subslot operator :=)
    • Note that while it's possible (not verified) we could avoid some rebuilds, this is both not guaranteed and doing the work now allows us to migrate to -compat on libxcrypt in future with a revbumped virtual (not yet possible due to so many pre-built binaries requiring compat).

Developer information

gyakovlev has implemented a QA check to ease finding missing dependencies.

  • If your package links against libcrypt, you need to explicitly depend on virtual/libcrypt:= in DEPEND and RDEPEND.
    • Please do revision bump (straight-to-stable) your package (recommended: git mv) when adding a either the dependency or a subslot operator for it. Revbump all versions in tree to avoid the need for stabilisation later and to apply the change consistently.
  • If your package doesn't link against libcrypt but uses it otherwise (e.g. runtime dlopen), please ensure you RDEPEND on virtual/libcrypt (no := required for rebuilds).
  • If your package doesn't use libcrypt, no action is required.

TODO

  1. Get sufficient testing on tinderboxes (ago, toralf, ...)
    • sam is going to switch to libxcrypt for arch testing too
  2. Ensure we're happy with the virtuals
  3. Discuss plan for migration (draft news item, ...)
  4. Achieve full set of keywords
    1. Finalise keywording in bug #749933 (as of writing, only ~m68k remains)
    2. Stabilise a version of libxcrypt on all architectures where we have stable glibc, ongoing in bug #796722

Migration plan

  1. Revbump sys-libs/glibc with IUSE="crypt" (not +crypt)
  2. Revbump sys-libs/libxcrypt with IUSE="+system"?
  3. Unmask the virtual
  4. Migrate masks to musl + uclibc-ng profiles? (... and include a force on the m68k prices until they keyword it?)

Future work

We can look into building sys-libs/libxcrypt with -compat in the distant future (and even then, not guaranteed), but this is not doable for now given the large number of binaries which expect sys-libs/glibc's libcrypt.

We can consider making another version of the virtual which is indefinitely masked for this. Please note that it's not possible to migrate back to sys-libs/glibc's libcrypt if you disable the compatibility interface in sys-libs/libxcrypt.

See also

Retrieved from "https://wiki.gentoo.org/index.php?title=Project:Toolchain/libcrypt_implementation&oldid=1043390"