Project:Portage/Sync

Changes
/etc/portage/repos.conf/* New setting for all repository types (needed): auto-sync = yes/no, true/false # default if absent: no/false

New for git sync-type: sync-depth = n where n = {0,1,2,3,...} 0 is equivalent to full git history 1 shallow clone, only current state (default if option is absent) 2, 3, ... history depth to download.

New sync-type modules: sync-type = svn # sync a subversion repository

sync-type = websync # Perform an emerge-webrsync operation on           the repo. By using this sync-type, a user can do a normal 'emerge --sync' or 'emaint sync -a' instead of an           'emerge-webrsync' command.

sync-type = laymanator # (if installed) runs layman's code to            sync the repository (overlay)

New native portage postsync hooks /etc/portage/postsync.d/* Runs all hooks found there once, after all repos have been synced. Compatible with all existing postsync.d scripts. Replaces the portage-utils installed post_sync hook script.

/etc/portage/repo.postsync.d/* Runs each script with three arguments: repo name, sync-uri, location Each script is run at the completion of each repository that was synced. Using these arguments, the hook script can decide if it needs to run or just exit. This is Useful for when only a certain repo has been updated and needs to trigger some other process or update system.

Migration
Edit all repos.conf/*.conf files, add the auto-sync option to each defined repository. For sync-type, edit it to one of the installed supported types. [gentoo] location = /usr/portage sync-type = rsync sync-uri = rsync://rsync.gentoo.org/gentoo-portage auto-sync = yes

For existing repos.conf/layman.conf file:

1) change/add the sync-type       [wtk]        location = /var/lib/layman/wtk        sync-type = laymanator        sync-uri = git://tremily.us/wtk-overlay.git        auto-sync = yes

2) Ensure you have the correct layman version installed with it's laymanator module also installed.

Alternate method:

If you are using the old make.conf layman conf_type:

1) Edit /etc/layman.layman.cfg add the repos.conf config type       conf_type : make.conf,repos.conf 2) Generate the new repos.conf/layman.conf file, run: layman-updater 3) Edit layman.cfg, remove 'make.conf' from the conf_type.       conf_type : repos.conf 4) Delete layman's make.conf file rm /var/lib/layman/make.conf 5) Edit make.conf, remove the source /var/lib/layman/make.conf line

Operation
Primary control of all sync operations has been moved from emerge to emaint. "emerge --sync" now runs the emaint sync module with the --auto option. This --auto option performs a sync on only those repositories with the auto-sync setting set to yes' or 'true'. If the auto-sync option is not set to yes or is absent, then emerge --sync may not sync any repositories.

The 'emaint sync' module operates similar to layman. It can sync single or multiple repos. It also syncs repos with 'auto-sync = no' set in the respective repos.conf file. This gives users and administrators more flexibility in how and when repositories are synced.

emaint sync options: -a, --auto     Sync auto-sync enabled repos only -A, --allrepos Sync all repos that have a sync-url defined -r REPO, --repo REPO Sync the specified repo --sync-submodule {glsa,news,profiles} Restrict sync to the specified submodule(s) (only rsync supported currently)

Examples
$ emaint sync -a

Equivalent to 'emerge --sync', sync all 'auto-sync = true' repos.

$ emaint sync -r foo

Will sync the foo repo (ignores auto-sync setting)

$ emaint sync --allrepos

Sync all repositories with a valid sync-type and sync-url defined. (ignores auto-sync setting)

Directory structure

 * /usr/lib/portage/
 * sync/
 * modules/
 * rsync/
 * __init__.py
 * rsync.py
 * git/
 * __init__.py
 * git.py
 * __init__.py
 * config_checks.py
 * controller.py
 * getaddrinfo_validate.py
 * old_tre_timestamp.py
 * syncbase.py

Moudule's __init__.py
Example module __init__.py definition

Validating repos.conf variables
Depending on the needs of the sync module, you can assign the "validate_config" variable in the module's spec the portage.sync.config_checks.CheckSyncConfig class or subclass it, adding any checks for additional variables needed to be defined in the repos.conf definition for that sync type.

Example module adding an additional validation test

Common classes used by sync modules

 * class SyncBase(object):........ Base Sync class for subclassing. It defines some common functions which can be used or overriden as needed. It also provides stubs for all functions needed by the controller.
 * def name:
 * def can_progressbar(self, func):
 * def __init__(self, bin_command, bin_pkg):
 * def _kwargs(self, kwargs):.... Sets internal variables from kwargs
 * def exists(self, **kwargs):.... Tests whether the repo actually exists
 * def sync(self, **kwargs):...... Sync the repository
 * def new(self, **kwargs):....... Stub only: Do the initial download and install of the repository
 * def _sync(self):............... Stub only: Update existing repository
 * def post_sync(self, portdb, location, emerge_config):.... Stub only: repo.sync_type == "Blank" NOTE: Do this after reloading the config, in case it did not exist prior to sync, so that the config and portdb properly account for its existence.


 * class CheckSyncConfig(object):..... Base repos.conf settings checks class
 * def __init__(self, repo=None, logger=None):.... Class init function
 * self.checks = ['check_uri', 'check_auto_sync']...... List of predefined checks to perform. When sublcassing append to it or redfine as needed
 * def repo_checks(self):......................... Perform all checks available
 * def check_uri(self):.............................. Check the sync_uri setting
 * def check_auto_sync(self):................. Check the auto_sync setting