Project:Portage/Sync

Introduction
Starting with sys-apps/portage-2.2.16, portage now has a new modular plug-in sync system. This new system will ease code maintenance as well as make it possible for third party modules to be installed. Many of the users will be learning about the repos.conf style changes that have been in portage for several years now for the first time. Portage has been migrating away from the PORTDIR and PORTDIR_OVERLAY variable settings for quite some time. Those settings are very limited in their capabilities as compared to the repos.conf style of repository configuration.

The new plug-in sync system is the next step in that migration. Using the old PORTDIR, PORTDIR_OVERLAY settings you could only list where the repository was. You could not specify other important attributes about that repository. The repos.conf style configuration allows for settings to be added on a per sync-type basis. Along with this new expandability and the plug-in sync system, it will be much easier to change to new repository syncing methods.

Changes

 * New auto-sync setting for all repository types (needed):
 * New for git sync-type: where n = {0,1,2,3,...}
 * 0 is equivalent to full git history
 * 1 shallow clone, only current state (default if option is absent)
 * 2, 3, ... history depth to download.
 * New sync-type modules:
 * Sync a subversion repository
 * Perform an emerge-webrsync operation on the repo. By using this sync-type, a user can do a normal emerge --sync or emaint sync -a instead of running the emerge-webrsync command.
 * (if installed) runs layman's code to sync the repository (overlay)
 * New native portage postsync hooks
 * Runs all hook scripts found once, after all repos have been synced. Compatible with all existing postsync.d scripts. Replaces the portage-utils installed post_sync hook script.
 * Runs each hook script found with three arguments: repo name, sync-uri, location Each script is run at the completion of each repository that was synced. Using these arguments, the hook script can decide if it needs to run or just exit. This is Useful for when only a certain repo has been updated and needs to trigger some other process or update system.
 * Runs each hook script found with three arguments: repo name, sync-uri, location Each script is run at the completion of each repository that was synced. Using these arguments, the hook script can decide if it needs to run or just exit. This is Useful for when only a certain repo has been updated and needs to trigger some other process or update system.

Migration
The primary reason for upgrading from make.conf style PORTDIR (deprecated, only there for backward compatibility with old tools, the repos.conf setting is used for it's value) and PORTDIR_OVERLAY (still functional, but deprecated) listings and the new repos.conf style definitions is functionality. The repos.conf method is a lot more flexible and powerful for controlling the repositories. The PORTDIR_OVERLAY listings are simple listing which are not capable of listing the options to control the new sync features and abilities, let alone other portage capabilities such as priority.

Portage configuration
If does not exist

Then edit the file for your installation and desired settings, continue as needed with the remaining migration instructions. Edit all files, add the auto-sync option to each defined repository. For sync-type, edit it to one of the installed supported types.

The supported sync types are currently:
 * 1) rsync
 * 2) git
 * 3) svn
 * 4) webrsync (equivalent to running emerge-webrsync separately) Note: changed from websync to webrsync for consistency and popular demand
 * 5) cvs
 * 6) laymansync (if installed by layman)

Example local overlay sync-able from git backup.

The above overlay can be synced with

Manually editing /etc/portage/repos.conf/layman.conf
For an existing repos.conf/layman.conf file:

1) change/add the sync-type

2) Ensure you have >=app-portage/layman-2.3.0 installed with it's laymansync module also installed.

layman-updater Method
Migrating from the old make.conf layman conf_type to repos.conf type:

1) Ensure you have the new plug-in sync portage version installed

2) Install the new laymansync enabled layman version. To install layman with the portage plugin, you must enable the sync-plugin-portage USE flag.

3) Edit layman.cfg change it to the repos.conf config type

save and exit the file.

4) Generate the new repos.conf/layman.conf file:

5) Delete layman's make.conf file

6) Edit /etc/portage/make.conf, remove the "source /var/lib/layman/make.conf" line

Done

Operation
Primary control of all sync operations has been moved from emerge to emaint. "emerge --sync" now runs the emaint sync module with the --auto option. This --auto option performs a sync on only those repositories with the auto-sync setting set to yes' or 'true'. If the auto-sync option is not set to yes or is absent, then emerge --sync may not sync any repositories.

The 'emaint sync' module operates similar to layman. It can sync single or multiple repos. It also syncs repos with 'auto-sync = no' set in the respective repos.conf file. This gives users and administrators more flexibility in how and when repositories are synced.

emaint sync options: -a, --auto     Sync auto-sync enabled repos only (includes defaulted repos) -A, --allrepos Sync all repos that have a sync-url defined -r REPO, --repo REPO Sync the specified repo --sync-submodule {glsa,news,profiles} Restrict sync to the specified submodule(s) (only rsync supported currently)

Examples
Equivalent to 'emerge --sync'. Sync all 'auto-sync = true' repos.

Sync the foo repo (ignores auto-sync setting)

Sync all repositories with a valid sync-type and sync-url defined. (ignores auto-sync setting)

--

Directory structure

 * /usr/lib/portage/
 * sync/
 * modules/
 * rsync/
 * __init__.py
 * rsync.py
 * git/
 * __init__.py
 * git.py
 * __init__.py
 * config_checks.py
 * controller.py
 * getaddrinfo_validate.py
 * old_tre_timestamp.py
 * syncbase.py

Moudule's __init__.py
Example module __init__.py definition

Validating repos.conf variables
Depending on the needs of the sync module, you can assign the "validate_config" variable in the module's spec the portage.sync.config_checks.CheckSyncConfig class or subclass it, adding any checks for additional variables needed to be defined in the repos.conf definition for that sync type.

Example module adding an additional validation test

Common classes used by sync modules

 * class SyncBase(object):........ Base Sync class meant to be subclassed. It defines some common functions which can be used or overriden as needed by the target module. It provides stubs for all functions needed by the main sync controller.
 * def name:
 * def can_progressbar(self, func):
 * def __init__(self, bin_command, bin_pkg):
 * def _kwargs(self, kwargs):.... Sets internal variables from kwargs
 * def exists(self, **kwargs):.... Tests whether the repo actually exists
 * def sync(self, **kwargs):...... Stub only: Sync the repository
 * def post_sync(self, portdb, location, emerge_config):.... Stub only: repo.sync_type == "Blank" NOTE: Do this after reloading the config, in case it did not exist prior to sync, so that the config and portdb properly account for its existence.


 * class NewBase(object):........ Base Sync class which subclasses SyncBase and also meant to be subclassed. It adds a predefined sync, and stubs for new and update. It forms the base for a module which requires separate new and update operations.
 * def sync(self, **kwargs):...... Conditionally syncs the repository using either the new or update functions
 * def new(self, **kwargs):....... Stub only: Do the initial download and install of the repository
 * def update(self):............... Stub only: Update existing repository


 * class CheckSyncConfig(object):..... Base repos.conf settings checks class
 * def __init__(self, repo=None, logger=None):.... Class init function
 * self.checks = ['check_uri', 'check_auto_sync']...... List of predefined checks to perform. When sublcassing append to it or redfine as needed
 * def repo_checks(self):......................... Perform all checks available
 * def check_uri(self):.............................. Check the sync_uri setting
 * def check_auto_sync(self):................. Check the auto_sync setting