User:NeddySeagoon/HOWTO Update Old Gentoo

From Gentoo Wiki
Jump to: navigation, search

HOWTO Update Old Gentoo

This is mostly for the educational experience and to write this document to show what can be done if you really really have to do it.

Important
Reinstalling a system several years old will be faster than this process

Background

Some time ago, it looks like mid 2011, my four or five discrete systems were retired in favour of a Gen7 HP Microserver running KVMs to replace the physical servers. The power savings paid for the Microserver in 18 months and its still going.

The poor wee thing struggled in only 8G of RAM and I wanted to run another KVM. I upgraded to 16G RAM, that's double the maximum its supposed to take but that's another HOWTO.

In 2011, one of the services moved to a KVM was my router. It was originally Smoothwall 3 on a 233MHz Pentium. To be able to test the KVM implementation I created a number of throw away KVMs for testing, before I exposed real hardware to the new router.

Of course, being throw away KVMs, they were never thrown away, they were just shut down and left.

Being in need of another KVM, I thought it might be fun to attempt to upgrade a 2011 Gentoo install, rather that install a new KVM.

Overview

This process is going through some of the missed updates. It does imply that nothing should break as a result of following this process, if you can get all the source files required for each step. That's a big if for systems more than three years out of date.

The Starting Point

The datestamp on the gentoo repo

user $cat /usr/portage/metadata/timestamp
Thu Aug 11 17:36:42 UTC 2011

Any time in 2019, a 2011 install qualifies as old and out of date

Profile

user $eselect profile list
 default/linux/amd64/10.0  
 default/linux/amd64/10.0/desktop  
 default/linux/amd64/10.0/desktop/gnome  
 default/linux/amd64/10.0/desktop/kde  
 default/linux/amd64/10.0/developer 
 default/linux/amd64/10.0/no-multilib  
 default/linux/amd64/10.0/server 
 hardened/linux/amd64  
 hardened/linux/amd64/selinux  
 hardened/linux/amd64/no-multilib  *
 hardened/linux/amd64/no-multilib/selinux 
 selinux/2007.0/amd64  
 selinux/2007.0/amd64/hardened  
 selinux/v2refpolicy/amd64  
 selinux/v2refpolicy/amd64/desktop 
 selinux/v2refpolicy/amd64/developer  
 selinux/v2refpolicy/amd64/hardened  
 selinux/v2refpolicy/amd64/server

binutils

Blue_Test ~ #binutils-config -l
  [1] x86_64-pc-linux-gnu-2.21.1 *

gcc

Blue_Test ~ #gcc-config -l
  [1] x86_64-pc-linux-gnu-4.5.3 *
  [2] x86_64-pc-linux-gnu-4.5.3-hardenednopie
  [3] x86_64-pc-linux-gnu-4.5.3-hardenednopiessp
  [4] x86_64-pc-linux-gnu-4.5.3-hardenednossp
  [5] x86_64-pc-linux-gnu-4.5.3-vanilla

Kernel

Blue_Test ~ #uname -a
 Linux Blue_Test 2.6.39-hardened-r8 #1 SMP Thu Jul 28 19:39:39 BST 2011 x86_64 AMD Phenom(tm) 9550 Quad-Core Processor AuthenticAMD GNU/Linux

World

It was a fairly minimal install for firewall testing

FILE /var/lib/portage/world
app-admin/metalog
app-admin/sudo
app-portage/gentoolkit
app-portage/ufed
app-text/wgetpaste
net-analyzer/tcpdump
net-misc/dhcpcd
net-misc/ntp
sys-apps/ethtool
sys-apps/portage
sys-apps/usermode-utilities
sys-boot/grub-static
sys-devel/gettext
sys-kernel/hardened-sources
sys-libs/gpm
sys-process/dcron

The process described here should work for any type of install. The ebuilds are available. The hard bit is finding old source code. Its a process document, not a step by step guide. This example does not have a desktop or even a window manager. Updating KDE or GNOME should be more of the same.

Plan A - Just Go For It

  • Install git
  • Move the old repo out of the way, so we can still refer to it for bits and pieces.
  • Get an up to date ::gentoo repo
Blue_Test ~ # emerge -uDNav world
Blue_Test ~ # etc-update

An eight year old portage with a current repo ...

Plan A - Implementation

Install git
Blue_Test ~ #emerge git -av
 
  * IMPORTANT: 2 news items need reading for repository 'gentoo'.
  * Use eselect news to read news items.
 
  * Last emerge --sync was 8y 21d 4h 12m 6s ago.
 
 These are the packages that would be merged, in order:
 
 Calculating dependencies... done!
 [ebuild   R    ] dev-vcs/git-1.7.6  USE="blksha1 curl iconv perl python threads webdav -bash-completion -cgi -cvs -doc -emacs -gtk (-ppcsha1) -subversion -tk -xinetd" 0 kB
 
 Total: 1 package (1 reinstall), Size of downloads: 0 kB
 
 Would you like to merge these packages? [Yes/No] 

Git is already installed. That was an unexpected bonus.

Shuffle repos

Move the old repo out of the way.

Blue_Test ~ #mv /usr/portage /usr/portage_old

Clone the git ::gentoo repo

Blue_Test ~ #mkdir /usr/portage
Blue_Test ~ #cd /usr/portage
Fix the Profile Definition

The current profile has been removed, so to put off dealing with profile updates change the /etc/make.profile symbolic link to point to the copy in the repo we saved earlier.

Note
This install predates the migration to /etc/portage/..
Blue_Test ~ #cd /etc
Blue_Test ~ #rm make.profile
Blue_Test ~ #ln -s ../usr/portage_old/profiles/hardened/linux/amd64/no-multilib make.profile

Plan A Summary

The old system is intact, with its original profile and a current ::gentoo repo.

In theory, in a perfect world, emerge -uDNav world just works.

Blue_Test ~ #emerge -uDNav world
 
 These are the packages that would be merged, in order:
 
 Calculating dependencies \ * Missing digest for '/usr/portage/sys-fs/sysfsutils/sysfsutils-2.1.0.ebuild'
  / * Manifest not found for '/usr/portage/virtual/pager/pager-0.ebuild'
  | * Manifest not found for '/usr/portage/virtual/ssh/ssh-0.ebuild'
 ...

The old portage sys-apps/portage-2.2.0_alpha50 knows nothing of the new manifest format, and lots of other things besides, so this approach ends here.

Plan B Use a git/CVS Repo

  • Move the old repo out of the way, so we can still refer to it for bits and pieces.
  • Use a git repo, to do easy checkouts by dates.
  • Fall back to a CVS repo if required
  • Incrementally update stage 1 by running /usr/portage/scripts/bootstrap.sh

/usr/portage/scripts/bootstrap.sh is the script used to do the stage 1 part of a stage 1 install. Its still in the git tree. Its not going to be quite that easy as there will be at least two profile changes along the way.

Plan B - Implementation

The first two steps were completed as part of Plan A. We need to see how far git goes back ... git checkout `git rev-list -n 1 --first-parent --before="2015-09-01" master` looks about the limit. Its not far enough.

Time to rinse and repeat with CVS.

First build CVS

Put the old Portage back.

Blue_Test ~ #emerge -1av cvs

installed dev-vcs/cvs-1.12.12-r9. It even found the source code.

Open port 2401 in your firewall.

Blue_Test ~ #cvs -d :pserver:anonymous@anoncvs.gentoo.org:/var/cvsroot co -D "2012-08-11 GMT" gentoo-x86

Now we have three copies of the ::gentoo

  • /usr/portage as it was in 2011
  • from git with all the git history
  • from CVS until it was migrated to git

That's all the ebuilds that were ever in the ::gentoo repository.

The CVS Tree

Getting closer

Blue_Test ~ #emerge -avuDN @world
 !!! All ebuilds that could satisfy "virtual/pkgconfig" have been masked.
 !!! One of the following masked packages is required to complete your request:
 - virtual/pkgconfig-0-r1::gentoo (masked by: EAPI 5) 
 
 The current version of portage supports EAPI '4'. You must upgrade to a
 newer version of portage before EAPI masked packages can be installed.

Portage understands the manifests now, just not the EAPI.

Lets go back in time and use a cvs checkout that is almost the same as our portage

Blue_Test ~ #cd /usr
Blue_Test ~ #cvs -d :pserver:anonymous@anoncvs.gentoo.org:/var/cvsroot co -D "2011-08-11 GMT" gentoo-x86
Blue_Test ~ #ln -s gentoo-x86 portage

This is just for testing. Using a cvs checkout as the repo is not quite as straight forward as as a git clone.

Once that works, go forward a year

Blue_Test ~ #cd /usr
Blue_Test ~ #cvs -d :pserver:anonymous@anoncvs.gentoo.org:/var/cvsroot co -D "2012-09-01 GMT" gentoo-x86

and do the update.

After rummaging around with help from Google, I found all the bits I needed to make the update run. It was quite a nostalgia trip, using a portage from 2011.

My system has moved forward a year. There have been no profile updates yet.

Plan C

This is a refinement of plan B and probably the right way.

Gentoo tries to keep an upgrade path for about a year, so lets try one year steps. Before you cross an EAPI boundary, you need a new portage that understands the new EAPI.

Overview

  • Get a suitable ebuild repo (CVS or git)
  • GENTOO_MIRRORS="" FEATURES="digest assume-digests" emerge -uDNavf world --keep-going to fetch sources.
  • Find missing sources or delete old ebuilds that won't be used ... or both.
  • emerge binutils gcc, then switch to the new versions.
  • GENTOO_MIRRORS="" FEATURES="digest assume-digests" emerge -uDNav world --keep-going to actually update.
  • perl-cleaner --all
  • python-updater
  • revdep-rebuild or emerge @preserved-rebuild
  • etc-update
  • read NEWS and act on it
  • update kernel
  • emerge --depclean (carefully)
  • reboot into new kernel
  • tidy up as required
  • Rinse and repeat at annual intervals.

Timesavers

Set GENTOO_MIRRORS="" That's not the same as leaving GENTOO_MIRRORS unset. The former instructs portage not to use the gentoo mirror system at all, the latter permits the use of a default set of mirrors.

Source code is removed from the gentoo mirror system at the same time as ebuilds are removed from the tree. This setting tells portage to use the SOURCE_URI list in the ebuild to locate source files without checking the mirrors first.

Important
This is normally regarded as poor netiquette, that is why Gentoo provides a mirror system in the first place. Revert this setting later

The NEWS System

Gentoo News items are normally distributed with emerge --sync. As this process uses cvs and git repos directly, other arrangements need to be made for reading news items.

What action, if any, needs to be taken in response to news is install specific.

Warning
At each step, act on any News items that apply to your install

Problems With the CVS Repro

The Attic

Many things will have been moved to the Attic. This changes the headers in the ebuilds but not the digests.

Blue_Test ~ #emerge -avuDN @world
 Calculating dependencies - * Digest verification failed:
  * /usr/gentoo-x86/sys-libs/glibc/glibc-2.15-r2.ebuild
  * Reason: Filesize does not match recorded size
  * Got: 8000
  * Expected: 7994

Notice the header # $Header: /var/cvsroot/gentoo-x86/sys-libs/glibc/Attic/glibc-2.15-r2.ebuild,v 1.29 2013/02/12 15:53:08 jer Exp $

Its grown an "/Attic" to make the ebuild 6 characters longer.

The choices are fix the ebuild headers or fix the digests. Fixing headers to match the existing digests is more secure. Fixing the digests is faster since portage offers FEATURES="digest"

Understand the security implications before you make your choice. I used the FEATURES="digest" method. My sed foo is far too weak to fix the ebuild headers.

Blue_Test ~ #GENTOO_MIRRORS="" FEATURES="digest assume-digests" emerge -1av portage
 FEATURES variable contains unknown value(s): fixpackages 
 
  * IMPORTANT: 2 news items need reading for repository 'gentoo'.
  * Use eselect news to read news items.
 
  * 
  * The FEATURES=digest setting can prevent corruption from being noticed.
  * The `repoman manifest` command is the preferred way to generate
  * manifests and it is capable of doing an entire repository or category at
  * once.
  *

Problems With Finding Source Files

The Repo Itself

Recreating all the digests for packages that portage wants to install requires all the sources for all the ebuilds in the ebuild leaf directory. That's fine if you have or can find all the sources.

In practice, any ebuild that will not be used can be removed. That limits source code finding to the absolute minimum.

https:// connections

Don't even think about it on an old system.

Source files that are on a SSL protected server are out of reach of on old system. Certificates have been changed and SSL protocols have changed. Download such source files via an intermediate system.

Trying is harmless.

Warts on My System

My system is 64 bit only. It boots using grub-static, which requires 32 bit support in the kernel to install. My kernels do not have 32 bit support so installing grub-static fails. Use portages --exclude="grub-static" for cases like this. More permanently, use package.mask

Other Pitfalls

In no particular order.

Its /etc/make.conf, not /etc/portage/make.conf

Old emerge --depclean is a bit aggressive. Its a very bad thing to let it rip out eselect-python as that breaks emerge. Fixing it is left as an exercise for the reader.

Don't forget about

perl-cleaner --all
python-updater
revdep-rebuild
etc-update
DEVTMPFS=y in the kernel
CPU_FLAGS_X86=
keeping the kernel up to date
switch from udev to eudev
sshd changed to prohibitpassword for root

Not all of these may apply to your case. You might even find a few more.

Progress

Using cvs

For testing and working out how to use a cvs repo as the ::gentoo tree.

cvs -d :pserver:anonymous@anoncvs.gentoo.org:/var/cvsroot co -D "2011-09-01 GMT" gentoo-x86


cvs -d :pserver:anonymous@anoncvs.gentoo.org:/var/cvsroot co -D "2012-09-01 GMT" gentoo-x86

The first step up. That was difficult. Finding sources and removing ebuilds that were not required were the problem areas.

cvs -d :pserver:anonymous@anoncvs.gentoo.org:/var/cvsroot co -D "2013-09-01 GMT" gentoo-x86

This was actually two steps as EAPI 5 was introduced. A portage that understands EAPI 5 was required to get to 2013-09-01 GMT.

The /10.0/ to /13.0/ profile change was here too.

cvs -d :pserver:anonymous@anoncvs.gentoo.org:/var/cvsroot co -D "2014-09-01 GMT" gentoo-x86

Switched to eudev.

Tidying Up

Depending on how much free space is available, this step may need to be performed more or less frequently. Its documented here as this is a good time, before moving to a git ::gentoo repo.

Remove old kernels from /usr/src
Remove /var/tmp/portage, or its contents.
eclean -d {packages distfiles}
Remove /var/log/portage

Moving to git

To free up space, the gentoo-x86 cvs repo can be removed.

Get the entire ::gentoo git repo.

root #cd /usr
root #git clone https://github.com/gentoo/gentoo.git ./gentoo-git

Fix the /usr/portage symlink to point to gentoo-git, rather than gentoo-x86. Thats the entire ::gentoo git repo as it was at the time of the clone. For me, the newest commit is Date: Mon Sep 9 12:51:11 2019 +0200 As we are still 5 years out of date, we can't use that as is.

Using git

The first commit (all of the cvs gentoo-x86) was Saturday Aug 8 2015. The next step is from git. That fits nicely with the annual Sep 1st steps.

root #git checkout `git rev-list -n 1 --first-parent --before="2015-09-01" master`

This puts the HEAD of the git repo at Date: Tue Sep 1 13:30:01 2015 +0200

The FEATURES="digest" is no longer required. Git does not have an attic.

As the repo path was changed from /usr/gentoo-x86 to /usr/gentoo-git

Fix /etc/portage/repos.conf

Fix /etc/portage/make.profile

If ssh password login for root is required, check sshd_config before the reboot.

2016
root #git checkout `git rev-list -n 1 --first-parent --before="2016-09-01" master`

Only sources for eight packages no longer available from the SOURCE_URI= in the ebuilds. Once you get to three years old, the GPL requires that sources can still be obtained, so it gets easier then. Obtained does not mean on line.

There in no need to detail the steps any more.

2017
root #git checkout `git rev-list -n 1 --first-parent --before="2017-09-01" master`
2018
root #git checkout `git rev-list -n 1 --first-parent --before="2018-09-01" master`
2019
root #git checkout `git rev-list -n 1 --first-parent --before="2019-09-01" master`
Up To Date
root #git checkout master
root #git pull

I still have the migration to the 17.1 profile to do.