User:Ajak/Portage Performance

From Gentoo Wiki
Jump to:navigation Jump to:search

Portage performance notes

I've noticed some odd things with regard to Portage performance. I'm documenting them here so I have a public way to reference them.

cpython vs pypy

mgorny has noticed that pypy3 is much faster for dependency calculations (for him). In his blog post, he notes that many people were not able to reproduce his results. I surmise that much of this is due to few people realizing that USE=native-extensions drastically affects pypy performance. With that flag on, Portage compiles C extensions, while's performance page notes, "Any C extension module recompiled with PyPy takes a very large hit in performance. PyPy supports C extension modules solely to provide basic functionality."

Once I turn that flag off, I can indeed produce a similar speedup:

$ hyperfine "EPYTHON=pypy3 emerge -pvuDN @world" "EPYTHON=python3.10 emerge -pvuDN @world"
Benchmark 1: EPYTHON=pypy3 emerge -pvuDN @world
  Time (mean ± σ):     33.539 s ±  1.081 s    [User: 32.602 s, System: 0.642 s]
    Range (min … max):   32.102 s … 35.993 s    10 runs

Benchmark 2: EPYTHON=python3.10 emerge -pvuDN @world
  Time (mean ± σ):     104.236 s ±  1.403 s    [User: 102.861 s, System: 0.694 s]
    Range (min … max):   102.330 s … 106.934 s    10 runs

  'EPYTHON=pypy3 emerge -pvuDN @world' ran
      3.11 ± 0.11 times faster than 'EPYTHON=python3.10 emerge -pvuDN @world'

However, even with this change, some people still report no significant change between cpython and pypy.

Anecdotally, I've noticed that package merges are much slower with pypy3 (as opposed to just dependency calculations as above). I haven't found a package with which I can reproducibly demonstrate the performance difference.