Project:Portage/Sync

Changes

 * New auto-sync setting for all repository types (needed):
 * New for git sync-type: where n = {0,1,2,3,...}
 * 0 is equivalent to full git history
 * 1 shallow clone, only current state (default if option is absent)
 * 2, 3, ... history depth to download.
 * New sync-type modules:
 * Sync a subversion repository
 * Perform an emerge-webrsync operation on the repo. By using this sync-type, a user can do a normal emerge --sync or emaint sync -a instead of running the emerge-webrsync command.
 * (if installed) runs layman's code to sync the repository (overlay)
 * New native portage postsync hooks
 * Runs all hook scripts found once, after all repos have been synced. Compatible with all existing postsync.d scripts. Replaces the portage-utils installed post_sync hook script.
 * Runs each hook script found with three arguments: repo name, sync-uri, location Each script is run at the completion of each repository that was synced. Using these arguments, the hook script can decide if it needs to run or just exit. This is Useful for when only a certain repo has been updated and needs to trigger some other process or update system.
 * Runs each hook script found with three arguments: repo name, sync-uri, location Each script is run at the completion of each repository that was synced. Using these arguments, the hook script can decide if it needs to run or just exit. This is Useful for when only a certain repo has been updated and needs to trigger some other process or update system.

Migration
Edit all files, add the auto-sync option to each defined repository. For sync-type, edit it to one of the installed supported types.

Example local overlay with git backup.

Can be synced with

For an existing repos.conf/layman.conf file:

1) change/add the sync-type

2) Ensure you have the correct layman version installed with it's laymanator module also installed.

Alternate method:

Migrating from the old make.conf layman conf_type to repos.conf type:

1) Ensure you have the new plug-in sync portage version installed

2) Install the new laymanator enabled layman version

3) Edit layman.cfg add the repos.conf config type

4) Generate the new repos.conf/layman.conf file:

5) Edit layman.cfg, remove 'make.conf' from the conf_type.

6) Delete layman's make.conf file

7) Edit /etc/portage/make.conf, remove the "source /var/lib/layman/make.conf" line

Done

Operation
Primary control of all sync operations has been moved from emerge to emaint. "emerge --sync" now runs the emaint sync module with the --auto option. This --auto option performs a sync on only those repositories with the auto-sync setting set to yes' or 'true'. If the auto-sync option is not set to yes or is absent, then emerge --sync may not sync any repositories.

The 'emaint sync' module operates similar to layman. It can sync single or multiple repos. It also syncs repos with 'auto-sync = no' set in the respective repos.conf file. This gives users and administrators more flexibility in how and when repositories are synced.

emaint sync options: -a, --auto     Sync auto-sync enabled repos only (includes defaulted repos) -A, --allrepos Sync all repos that have a sync-url defined -r REPO, --repo REPO Sync the specified repo --sync-submodule {glsa,news,profiles} Restrict sync to the specified submodule(s) (only rsync supported currently)

Examples
Equivalent to 'emerge --sync'. Sync all 'auto-sync = true' repos.

Sync the foo repo (ignores auto-sync setting)

Sync all repositories with a valid sync-type and sync-url defined. (ignores auto-sync setting)

--

Directory structure

 * /usr/lib/portage/
 * sync/
 * modules/
 * rsync/
 * __init__.py
 * rsync.py
 * git/
 * __init__.py
 * git.py
 * __init__.py
 * config_checks.py
 * controller.py
 * getaddrinfo_validate.py
 * old_tre_timestamp.py
 * syncbase.py

Moudule's __init__.py
Example module __init__.py definition

Validating repos.conf variables
Depending on the needs of the sync module, you can assign the "validate_config" variable in the module's spec the portage.sync.config_checks.CheckSyncConfig class or subclass it, adding any checks for additional variables needed to be defined in the repos.conf definition for that sync type.

Example module adding an additional validation test

Common classes used by sync modules

 * class SyncBase(object):........ Base Sync class meant to be subclassed. It defines some common functions which can be used or overriden as needed by the target module. It provides stubs for all functions needed by the main sync controller.
 * def name:
 * def can_progressbar(self, func):
 * def __init__(self, bin_command, bin_pkg):
 * def _kwargs(self, kwargs):.... Sets internal variables from kwargs
 * def exists(self, **kwargs):.... Tests whether the repo actually exists
 * def sync(self, **kwargs):...... Stub only: Sync the repository
 * def post_sync(self, portdb, location, emerge_config):.... Stub only: repo.sync_type == "Blank" NOTE: Do this after reloading the config, in case it did not exist prior to sync, so that the config and portdb properly account for its existence.


 * class NewBase(object):........ Base Sync class which subclasses SyncBase and also meant to be subclassed. It adds a predefined sync, and stubs for new and update. It forms the base for a module which requires separate new and update operations.
 * def sync(self, **kwargs):...... Conditionally syncs the repository using either the new or update functions
 * def new(self, **kwargs):....... Stub only: Do the initial download and install of the repository
 * def update(self):............... Stub only: Update existing repository


 * class CheckSyncConfig(object):..... Base repos.conf settings checks class
 * def __init__(self, repo=None, logger=None):.... Class init function
 * self.checks = ['check_uri', 'check_auto_sync']...... List of predefined checks to perform. When sublcassing append to it or redfine as needed
 * def repo_checks(self):......................... Perform all checks available
 * def check_uri(self):.............................. Check the sync_uri setting
 * def check_auto_sync(self):................. Check the auto_sync setting