User:Csfore/Portage sync eselect repository update
Starting with Portage version 2.2.16, Portage now has a modular, plug-in sync system. This new sync system will ease code maintenance as well as make it possible for third party modules to be installed. Many of the users will be learning about the repos.conf style changes that have been in Portage for several years now for the first time.
Portage has been migrating away from make.conf's PORTDIR and PORTDIR_OVERLAY variable settings for quite some time. Those settings are very limited in their capabilities as compared to the repos.conf style of repository configuration. The new sync system is the next step in that migration.
Using PORTDIR and PORTDIR_OVERLAY in make.conf limited Portage by only defining where the repository was located. Portage had no controls to specify other important attributes about that repository. The repos.conf configuration style allows for settings to be defined on a per repository, per sync-type basis. Along with this new expandability and the plug-in sync system, it will be much easier to change to new repository syncing methods.
General help
While Portage can handle repos.conf as either a file or a directory of files. The preferred method is to be used as a directory. Other tools like layman and mirrorselect require and expect it to be a directory. Layman creates and manages its own layman.conf file to register installed overlays with Portage. mirrorselect looks for a repos.conf/gentoo.conf file in order to modify the
sync-uri
parameter for the gentoo repository.Changes
- /etc/portage/repos.conf/*
- New auto-sync setting for all repository types (needed):
auto-sync = yes # default, Possible values are: yes, no, true, false
- New for git sync-type:
sync-depth = n
where n = {0,1,2,3,...}- 0 is equivalent to full git history
- 1 shallow clone, only current state (default if option is absent)
- 2, 3, ... history depth to download.
- New sync-type modules:
sync-type = svn
Sync a subversion repository- for <=sys-apps/portage-2.2.18
sync-type = websync
- for >=sys-apps/portage-2.2.19
sync-type = webrsync
Perform an emerge-webrsync operation on the repo. By using this sync-type, a user can do a normal emerge --sync or emaint sync -a instead of running the emerge-webrsync command. sync-type = laymansync
(if installed) runs layman's code to sync the repository (overlay)
- New native portage postsync hooks
- /etc/portage/postsync.d/*
Runs all hook scripts found once, after all repos have been synced. Compatible with all existing postsync.d scripts. Replaces the portage-utils installed post_sync hook script. - /etc/portage/repo.postsync.d/*
Runs each hook script found with three arguments: repo name, sync-uri, location
Each script is run at the completion of each repository that was synced. Using these arguments, the hook script can decide if it needs to run or just exit. This is useful for when only a certain repo has been updated/synced and needs to trigger some other process or update system.
- /etc/portage/postsync.d/*
- New auto-sync setting for all repository types (needed):
Migration
The primary reason for upgrading from make.conf style PORTDIR (deprecated, only there for backward compatibility with old tools, the repos.conf setting is used for its value) and PORTDIR_OVERLAY (still functional, but deprecated) listings and the new repos.conf style definitions is functionality. The repos.conf method is a lot more flexible and powerful for controlling the repositories. The PORTDIR_OVERLAY listings are simple listing which are not capable of listing the options to control the new sync features and abilities, let alone other portage capabilities such as priority.
Portage configuration
If the /etc/portage/repos.conf directory does not exist:
root #
mkdir /etc/portage/repos.conf
The next step is only needed in order to change the default settings for the gentoo ebuild repository
root #
cp /usr/share/portage/config/repos.conf /etc/portage/repos.conf/gentoo.conf
Then edit the file to the desired settings, continue as needed with the remaining migration instructions. Edit all repos.conf/*.conf files, add the auto-sync option to each defined repository.
For sync-type, set it to one of the installed supported types.
Current supported sync types include:
rsync
git
svn
webrsync
orwebsync
(equivalent to running emerge-webrsync separately).[1]cvs
laymansync
(if installed by Layman)
[gentoo]
location = /var/db/repos/gentoo
sync-type = rsync
sync-uri = rsync://rsync.gentoo.org/gentoo-portage
auto-sync = yes
Example local overlay sync-able from a git remote:
[the-pit]
location = /var/db/repos/the-pit
sync-type = git
sync-uri = git://me.org/me/the-pit.git
auto-sync = no
The above overlay can be synced with
user $
emaint sync -r the-pit
Operation
Primary control of all sync operations has been moved from emerge to emaint. emerge --sync now runs the emaint sync module with the --auto
option. This --auto
option performs a sync on only those repositories with the auto-sync setting set to yes
or true
. If the auto-sync option is not set to yes
or is absent, then emerge --sync may not sync any repositories.
As a result of the default auto-sync = True/Yes setting, commands like eix-sync, esync -l, emerge --sync && layman -S will cause many repositories to be synced multiple times in a row. Please adjust configuration files or scripts as necessary for the new operation.
Due to the above default. For any repositories that should not be synced when emerge --sync or emaint sync --auto is run, be sure to set to set the
auto-sync = no
key/value on each repository.The emaint sync module operates similar to layman. It can sync single or multiple repos. It can also sync repos with auto-sync = no
set in the respective repos.conf file. This gives users and administrators more flexibility in how and when repositories are synced:
root #
emaint sync
emaint sync options: -a, --auto Sync auto-sync enabled repos only (includes defaulted repos) -A, --allrepos Sync all repos that have a sync-url defined -r REPO, --repo REPO Sync the specified repo --sync-submodule {glsa,news,profiles} Restrict sync to the specified submodule(s) (only rsync supported currently)
Examples
root #
emaint sync -a
Equivalent to emerge --sync. Sync all repositories where auto-sync = true
is set.
Due to the default
auto-sync = true
setting, this command will sync all repositories that do not have auto-sync = no
explicitly set.root #
emaint sync -r foo
Sync the foo repo (ignores auto-sync setting):
root #
emaint sync --allrepos
Sync all repositories with a valid sync-type and sync-url defined. (ignores auto-sync setting)
Plug-in sync API
Module specifications
Directory structure
- /usr/lib/portage/
- sync/
- modules/
- rsync/
- __init__.py
- rsync.py
- git/
- __init__.py
- git.py
- rsync/
- __init__.py
- config_checks.py
- controller.py
- getaddrinfo_validate.py
- old_tre_timestamp.py
- syncbase.py
- modules/
- sync/
Module's __init__.py
Example module __init__.py definition:
# Copyright 2014 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
"""Git plug-in module for portage.
Performs a git pull on repositories
"""
from portage.sync.config_checks import CheckSyncConfig
module_spec = {.
'name': 'git',
'description': __doc__,
'provides':{
'git-module': {
'name': "git",
'sourcefile': "git",
'class': "GitSync",
'description': __doc__,
'functions': ['sync', 'new', 'exists'],
'func_desc': {
'sync': 'Performs a git pull on the repository',
'new': 'Creates the new repository at the specified location',
'exists': 'Returns a boolean of whether the specified dir ' +
'exists and is a valid Git repository',
},
'func_parameters': {
'kwargs': {
'type': dict,
'description': 'Standard python **kwargs parameter format' +
'Please refer to the sync modules specs at ' +
'"https://wiki.gentoo.org:Project:Portage" for details',
},
},
'validate_config': CheckSyncConfig,
}
}
}
Definition: "module_spec"
Field Name | Type | Comments |
---|---|---|
name | string | The name of the module The name of the file providing the exported classes |
description | string | Typically is a pointer to the files docstring |
provides : [module1] |
dictionary dictionary |
Dictionary of of the different sub modules it provides. Note the field identifiers must be unique, but are not used outside of the plugin system. Unique module identifier. |
Definition: module_spec["provides"]["module1"]
Field Name | Type | Comments |
---|---|---|
name | string | Unique identifier of the exported class. |
sourcefile | string | The filename that contains the class without the ".py" extension |
class | string | the id/name of the class exported |
description | string | Description of the service it performs |
functions | list | The list of commands it exports |
func_desc | dictionary, its keys are the same as the 'functions' list members above | strings describing each of the exported functions |
validate_config | pointer | Assign it the CheckSyncConfig derived class used to validate repos.conf settings for this repo type. For simple modules, the base CheckSyncConfig is suitable. For others please refer to Validating repos.conf variables section of this page. |
Validating repos.conf variables
Depending on the needs of the sync module, the "validate_config" variable can be assigned in the module's spec the portage.sync.config_checks.CheckSyncConfig class or subclass it, adding any checks for additional variables needed to be defined in the repos.conf definition for that sync type.
Example module adding an additional validation test
# Copyright 2014 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
"""CVS plug-in module for portage.
Performs a cvs up on repositories
"""
from portage.localization import _
from portage.sync.config_checks import CheckSyncConfig
from portage.util import writemsg_level
class CheckCVSConfig(CheckSyncConfig):
def __init__(self, repo, logger):
CheckSyncConfig.__init__(self, repo, logger)
self.checks.append('check_cvs_repo')
def check_cvs_repo(self):
if self.repo.sync_cvs_repo is None:
writemsg_level("!!! %s\n" %
_("Repository '%s' has sync-type=cvs, but is missing sync-cvs-repo attribute")
% self.repo.name, level=self.logger.ERROR, noiselevel=-1)
module_spec = {
'name': 'cvs',
'description': __doc__,
'provides':{
'cvs-module': {
'name': "cvs",
'class': "CVSSync",
'description': __doc__,
'functions': ['sync', 'new', 'exists'],
'func_desc': {
'sync': 'Performs a cvs up on the repository',
'new': 'Creates the new repository at the specified location',
'exists': 'Returns a boolean of whether the specified dir ' +
'exists and is a valid CVS repository',
},
'func_parameters': {
'kwargs': {
'type': dict,
'description': 'Standard python **kwargs parameter format' +
'Please refer to the sync modules specs at ' +
'"https://wiki.gentoo.org:Project:Portage" for details',
'required-keys': ['options', 'settings', 'logger', 'repo',
'xterm_titles', 'spawn_kwargs'],
},
},
'validate_config': CheckCVSConfig,
}
}
}
Common classes used by sync modules
- class SyncBase(object):........ Base Sync class meant to be subclassed. It defines some common functions which can be used or overriden as needed by the target module. It provides stubs for all functions needed by the main sync controller.
- def name():
- def can_progressbar(self, func):
- def __init__(self, bin_command, bin_pkg):
- def _kwargs(self, kwargs):.... Sets internal variables from kwargs
- def exists(self, **kwargs):.... Tests whether the repo actually exists
- def sync(self, **kwargs):...... Stub only: Sync the repository
- def post_sync(self, portdb, location, emerge_config):.... Stub only: repo.sync_type == "Blank" NOTE: Do this after reloading the config, in case it did not exist prior to sync, so that the config and portdb properly account for its existence.
- class NewBase(object):........ Base Sync class which subclasses SyncBase and also meant to be subclassed. It adds a predefined sync(), and stubs for new() and update(). It forms the base for a module which requires separate new and update operations.
- def sync(self, **kwargs):...... Conditionally syncs the repository using either the new or update functions
- def new(self, **kwargs):....... Stub only: Do the initial download and install of the repository
- def update(self):............... Stub only: Update existing repository
- class CheckSyncConfig(object):..... Base repos.conf settings checks class
- def __init__(self, repo=None, logger=None):.... Class init function
- self.checks = ['check_uri', 'check_auto_sync']...... List of predefined checks to perform. When subclassing append to it or redfine as needed
- def repo_checks(self):......................... Perform all checks available
- def check_uri(self):.............................. Check the sync_uri setting
- def check_auto_sync(self):................. Check the auto_sync setting
- def __init__(self, repo=None, logger=None):.... Class init function
- ↑ Renamed from websync to webrsync in Portage 2.2.19 and higher for consistency and popular demand.