User:Pietinger/Tutorials/Optimize compile times
Even though this page is in the user namespace, corrections and additions are much appreciated! This is simply wiki policy, this page can be moved to the main wiki as soon as it achieves critical mass more.
I wrote this article so that I can link to it in the Gentoo forums if users have questions about it.
Optimize compile times
We have already two Wiki articles about MAKEOPTS and EMERGE_DEFAULT_OPTS. Why another tutorial ?
I will explain again the usage of these parameters because we have still threads with questions about it. Let us start with a simple configuration:
What does MAKEOPTS="-j8"
As you surely already know you can set the number of compile threads for a single package. It has two limitations:
1. The quantity of your CPU cores. SMT / Hyperthreading can be added if you have allowed it in your kernel configuration (default is "enabled"). I have an Intel i7 with 4 physical cores. Hyperthreading (this is the name from Intel for SMT) gives me 8 logical cores. So, I can use "-j8" as a maximum. Higher values would make all compiles SLOWER.
2. The quantitiy of your installed Memory (RAM). Every compile thread needs RAM for its work. But not every package use the same amount of RAM when compiling it. We have "small" packages which uses only less than 1 GB for every compile job. So, compiling this one with 8 threads would need only less than 8 GB of RAM. We have "monster" packages which needs approximately 4 GB RAM for every compile job; this was true for "rust", but with the actual version of "rust" I dont see this behavior anymore. If you emerge these kind of packages you would need 32 GB RAM for 8 compile jobs. If you have only 16 GB RAM installed you will get an OOM abort - Out of memory ! So, Gentoo has a recommendation:
Calculate 2 GB RAM for every compile job !
If you have 32 GB RAM the maximum would be "-j16" ... BUT ...
You can use only the smaller maximum of both limitations !
Examples:
4 cores and 16 GB RAM -> Use "-j4" because of your CPU
8 cores and 16 GB RAM -> Use "-j8"
8 cores and 32 GB RAM -> Use "-j8" because of your CPU
16 cores and 16 GB RAM -> Use "-j8" because of your RAM
16 cores and 32 GB RAM -> Use "-j16"
12 cores and 32 GB RAM -> Use "-j12" because of your CPU
How to handle "monster" packages ?
You have two choices:
1. You can lower MAKEOPTS so every compile will have 4 GB RAM for every compile job (divide your RAM with 4). Please dont laugh, but I use this. Why ? Because I do all my @world updates in the night and I dont wait for it. So, I dont care if it needs 3 hours or 4 hours ... But if you wait for it you can use:
2. Set exceptions for some big packages with /etc/portage/package.env
Here is an example you might use:
root #mkdir /etc/portage/env
root #nano -w /etc/portage/env/monster.conf
/etc/portage/env/monster.confMAKEOPTS="-j3"
root #nano -w /etc/portage/package.env /etc/portage/package.envapp-office/libreoffice monster.conf
dev-lang/rust monster.conf
dev-qt/qtwebengine monster.conf
sys-devel/gcc monster.conf
Using a tmpfs for Portage's TMPDIR
If you use additionally Portage_TMPDIR_on_tmpfs you must calculate a higher RAM usage. Most packages will need only 1 or 2 GB when compiling. Other packages will need up to 20 GB when compiling them. The question will be now: Shall I give my RAM to my compile jobs or using it for my TMPFS ? In most cases you will be faster when using your RAM for compiling. This means you must make also exceptions for packages using much RAM when compiling them. At minimum you should make a exception for rust:
root #mkdir -p /var/tmp/notmpfs
root #chown portage:portage /var/tmp/notmpfs
root #nano -w /etc/portage/env/notmpfs.conf
/etc/portage/env/notmpfs.confPORTAGE_TMPDIR="/var/tmp/notmpfs"
Now you can add it to your existing exception file:
root #nano -w /etc/portage/package.env /etc/portage/package.envapp-office/libreoffice monster.conf
dev-lang/rust monster.conf notmpfs.conf
dev-qt/qtwebengine monster.conf
sys-devel/gcc monster.conf
Instead using a tmpfs you can try also this options - coming from Tlhonmey (copied here from discussion page):
With regard to using tmpfs for compile, I find that as long as the system is reasonably stable and not going to crash it's simpler to temporarily set the following via sysctl:
- vm.dirty_ratio=90
- vm.dirty_background_ratio=85
- vm.dirty_writeback_centisecs=60000 #ten minutes is a reasonable compromise.
This allows the system to keep written files in the disk cache much longer, so the files created during the compile process won't ever touch the disk if you have sufficient memory and the compile is faster than the writeback timeout. The install is still safe since it calls sync. If you run short on memory the system will automatically push things out of the cache to make space. So you get most of the speed improvement of using a tmpfs, without the hassle of doing the memory management yourself. Please read also the discussion about this solution in our forum: https://forums.gentoo.org/viewtopic-t-1167080.html
Using EMERGE_DEFAULT_OPTS
As you surely already know you can set the number of how many competitive packages shall be compiled at the same time. Of course this will only work if you want emerge more than one single package; e.g. when you do an "emerge -u @world". With an EMERGE_DEFAULT_OPTS="--jobs 2" you tell portage to compile the first package of your @world list and at the same time the 2nd package of your @world list. This means it can double now the needed Ressources - RAM and threads. Why you want do this ? The answer is:
Every emerge of a package has different phases. Before a package will be compiled there is a configuration phase. And this configuration phase will ALWAYS use only ONE thread.
If you have two packages for an emerge and you have a MAKEOPTS="-j8" then your system does:
1. Configuring Package 1 with 1 thread working
2. Compiling Package 1 with 8 threads working
3. Configuring Package 2 with 1 thread working
4. Compiling Package 2 with 8 threads working
If you have two (or more) packages for an emerge and you have a MAKEOPTS="-j4" AND EMERGE_DEFAULT_OPTS="--jobs 2" then your system does:
1. Configuring Package 1 and Package 2 with 1 thread for each == 2 threads working
2. As soon as one of them is configured it starts compiling one of them == 5 threads working
3. As soon as the 2nd Package is configured it starts compiling this one also == 8 threads working
4. As soon as one of them is finished it starts a 3rd package - OR - it will compile the 2nd package with 4 threads working
As you can imagine the last situation CAN be faster ... BUT ... it could be also slower if you have one small package AND a 2nd BIG package, because the small package is finished in some seconds and you have only 4 compile threads for your 2nd big package.
So, yes it depends what you want to emerge !
If you want emerge (update) only ONE big package - like libreoffice - with "emerge -1uvD libreoffice" then you will be faster if you set MAKEOPTS to its maximum (see above) and DONT set EMERGE_DEFAULT_OPTS.
If you want to do an "emerge -u @world" with hundreds of packages then using EMERGE_DEFAULT_OPTS can speed up the complete Update (but dont compare the time for compiling a single package with the time needed if you update this package alone with a higher MAKEOPTS).