From Gentoo Wiki
Jump to:navigation Jump to:search

This is where we will flesh out ideas for the direction to take areas of portage's code.

Portage sync

This proposal is now fully implemented. Please refer to the Project:Portage/Sync wiki page for it's current specification.

This is the proposed layout of a totally revamped sync system. It uses the plug-in manager already in portage.


  1. Flexible
  2. Easily extended
  3. Less maintenance, core sync manager does not need changes for new modules
  4. Add on modules can be user installed for special types
The basic modular sync code is working, but please remember only minimal changes were done to the existing code in order to make it work. There is still much re-factoring to do to optimize the code for the new system. It will be open for code review at a later time. Please restrict code review to the basic design algorithm.

Proposed To Do

  1. Determine the information the module_spec needs to provide to the sync manager and clients <== InProgress
  2. Move the plug-in manager code out of emaint's namespace, into portage's
  3. Put a freeze on the current sync code to reduce the rebasing needed for the merge on completion <== Done
  4. Create a new feature branch to use for this change <== Done, plugin-sync
  5. Code the sync manager code <== Preliminary coding done.
  6. Determine the method to be able to do concurrent repo syncs
  7. Possibly move the TaskHandler class into the SyncManager class or subclass it
  8. Make the sync manager provide any common services functions such as dropping from root down to user privileges <== Done
  9. Move the sync code to a new rsync module <== Done
  10. Move the git code to it's own module <== Done
  11. re-write the code in action_sync(). <== minimally migrated to the new SyncManager class
  12. Begin testing, debugging <== Rsync, git tested working. CVS untested
  13. Add to module_spec for repos.conf variables it accepts/requires
  14. Add repos.conf validation checks functions
  15. Determine the location/directory for 3rd party sync modules to be installed to.
  16. Re-factor the code to optimize for the new structure.
  17. Move the web-rsync code from emerge into a module
  18. Modify emerge-webrsync cli to use the new webrsync module
  19. MORE testing, debugging.
  20. merge the branch back into master


CODE from the new rsync plug-in module. (Very basic)
# Copyright 2014 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

"""Rsync plug-in module for portage.
   Performs rsync transfers on repositories

module_spec = {
	'name': 'rsync',
	'description': __doc__,
		'module1': {
			'name': "rsync",
			'class': "RsyncSync",
			'description': __doc__,
			'functions': ['sync',],
			'func_desc': {'sync': 'Performs rsync transfers on the repository'}

Example Proposed extensions

CODE for a new extended git plug-in module
# Copyright 2014 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

"""Rsync plug-in module for portage.
   Performs a git operations on repositories

module_spec = {
	'name': 'git',
	'description': __doc__,
		'module1': {
			'name': "git",
			'class': "GitSync",
			'description': __doc__,
			'functions': ['sync', 'install', 'validate',],
			'func_desc': {
                            'sync': 'Performs a git pull on the repository',
                            'install': 'Installs a new git clone of the repository',
                            'validate': 'performs validation checks on the repos.conf ' \
                                        'entries for the repository',
                         'conf': {
                             'location': 'Repo location eg: /var/lib/gentoo/repos/gentoo',
                             'sync-type': 'The sync module type associated with the repo.  eg: "git"',
                             'sync-uri': 'the uri to use for initial installation',


portage-dev mail list threads

  • topic: New proposed modular sync system
  1. gmane thread

To be discussed at the next portage team meeting, Feb. 9, 2014 @ 19:00 UTC in #gentoo-meetings

Build environment plugins


Portage is getting overcomplex with growing number of build-env quirks and FEATURES. Would be good to figure out a nice plugin API to handle them.

Potential build-env plugins

The following features could be split into separate plugins:

  • ccache,
  • distcc,
  • fakeroot,
  • fuse-sandbox (in TODO),
  • ipc-sandbox,
  • network-sandbox,
  • sandbox,
  • userpriv.

Required facilities

Based on the potential plugins, the plugin system needs to provide at least:

  • entry/exit hooks for phase execution,
  • environment manipulation,
  • convenient wrapper creation and PATH manipulation,
  • ordering facilities — i.e. plugins requiring root privileges would have to be ordered before userpriv.