Project:Overlays/Internal Overlays team guide

This guide outlines the basic procedures for doing Overlays team work.

User requests

All user requests should be handled via Bugzilla, with specific details being logged in bug comments.

If users send requests via e-mail, please ask them to file a bug instead. This mean having a Gentoo Bugzilla account is obligatory.

If owner no longer has access to his original Bugzilla account, attempt to verify his identity (e.g. GitHub account owning the repository). If he is no longer interested in participating and requests removing the repository, this request can be handled by e-mail.

Adding external repositories

Before adding a repository

Before adding a new repository, please verify at least the following:

That it has profiles/repo_name file matching requested repository name (and not colliding with any existing repositories):
- If repo_name differs from requested name, ask user if he prefers updating repo_name or adding with the current name from repo_name,
- If repo_name collides with existing repository and the repository is unmaintained, you may propose to take over the name if the original owner agrees or doesn't respond in reasonable time.
That it has metadata/layout.conf specifying appropriate masters= entry.
- If masters is empty, ensure that the repository can work stand-alone (I.e. doesn't depend on packages or eclasses from ::gentoo),
- If masters specifies additional repositories, make sure they are all in repositories.xml.
That there are no obvious failures ;-),
That the user requesting the repository is its maintainer. If he's not, there are two possibilities:
- The owner needs to have a Bugzilla account and agree to adding him as the owner (i.e. having all bugs assigned to him),
- The owner needs to agree on being proxied by the requesting person, and the requesting person is then responsible for proxying all the bugs and other requests.
That the user doesn't request another e-mail for the owner than for Bugzilla account:
- If he does, get a confirmation that he either needs to have that registered on Bugzilla as well, or that he is fine with using his Bugzilla e-mail.

If repositories.xml is provided, make sure it is correct and update it as appropriate (see below). Otherwise, just write it from scratch.

Merging api.g.o pull requests

Since the GitHub mirror of api repository is one-way only, you need to merge all pull requests manually.

To clone the repositories and add GitHub as an additional remote:

user $git clone git@git.gentoo.org:data/api.git

user $cd api

user $git remote add github git@github.com:gentoo/api-gentoo-org.git

user $git config --add remote.github.fetch '+refs/pull/*/head:refs/remotes/github/pull/*'

Afterwards, to update pull requests:

user $git fetch github

The pull requests will be available as refs/github/pull/NN, i.e.:

user $git log -p github/pull/NN

user $git cherry-pick -S master..github/pull/NN

user $git commit --amend -S

This will cherry-pick all the commits from the PR and start amending the commit message of the latest. You need to add the Closes tag for GitHub there, i.e. add:

Closes: https://github.com/gentoo/api-gentoo-org/pull/NN

Adding to repositories.xml

The repositories.xml file is stored in data/api.git repository (files/overlays/repositories.xml). All Gentoo developers and staffers have write access (i.e. you may ask another developer to add his repository himself).

user $git clone git@git.gentoo.org:data/api.git

After updating the file, verify it:

user $make -C files/overlays check

Commit with OpenPGP signature, start commit message with repositories:, add a bug no and push (--signed).

Few tips for adding repositories.xml entries:

Always use <name/> from the repo_name file (otherwise, there will be a mismatch post adding),
Sort repositories lexically by name,
All <owner/> entries (there can be more than one) have to match valid Bugzilla accounts,
Sources:
- https:// is the preferred URI as it is secured and reachable behind restricted firewalls,
- Avoid ssh or make it last choice as it may trigger undesired password queries.

Updating and removing from repositories.xml

Similar rules as when adding new entries apply.

Usually modifications and removals should be done on request of owners, or with confirmations from them.

Exceptions are:

If repo_name changes and owner doesn't respond to the bug in reasonable time, you can update <name/> to match it (remember to reorder the entries!),
If there are problems with the repository or it is long-unmaintained, and the owner doesn't respond to the bugs, you can remove it,
If there are major issues with the repository doing harm to users (major security problems, damage to systems), you can remove it without waiting for response (and re-add when the issue is fixed).

When removing repositories, it would be nice to make sure we don't remove masters of other repositories. However, there is no good infrastructure to test this beforehand, so we may just wait for repo-mirror-ci to complain about it afterwards.

Hosting repositories on git.gentoo.org

Needed info

Before starting to add a new repository, make sure that the user has provided the necessary details. In particular:

repository name in appropriate namespace (repo/user for user repositories, repo/dev for dev repositories, repo/proj for project repositories),
his SSH key if he hasn't provided one for git.g.o before.

gitolite-admin repository

The gitolite-based git hosting is controlled via files in special gitolite-admin repository. Only Infra and Overlays team members are allowed to access it. Overlays team members have partial write access.

user $git clone git@git.gentoo.org:gitolite-admin.git

After altering appropriate files, commit the changes and push them. In response to the push, gitolite will print status messages, including any possible issues.

Adding SSH key for the user

The SSH key should be placed in keys/nondevs directory, with name formed of owner's e-mail address, followed by @00.pub (or successive numbers if more than one key is added).

For example, for user foo@example.com:

user $vim keys/nondevs/foo@example.com@00.pub

The file format consists of header-comments, followed by the key:

FILE keys/nondevs/foo@example.com@00.pub

# git-realname: Foo Bär
# git-realname-ascii: Foo Bar
# git-email: foo@example.com
# git-username: foo                                                     
ssh-... ...

git-realname specifies the user's real name (can be some nickname if he doesn't want to reveal it),
git-realname-ascii is the above transliterated to pure ASCII. Use it only if git-realname has non-ASCII characters.
git-email is the e-mail address.
git-username is some username. I have no clue what it does.

Don't forget to add it:

user $git add keys/nondevs/foo@example.com@00.pub

After adding the SSH key, run the cleanup script to update groups:

user $./scripts/clean.pl

Adding the repository

All repositories should be added to an appropriate conf/repo-*.conf file for the namespace of the repository:

user $vim conf/repo-user.conf

A new repository entry should look like:

FILE conf/repo.conf

repo repo/user/foo
	# gentoo-is-overlay = True
	# gentoo-terms-status = pending, accept on push
	RW+ = foo@example.com
	R   = @all daemon gitweb
	category = repo/user
	owner = "Foo Bar <foo@example.com>"
	desc = "My foo-ish repository"

RW+ controls admin access, RW write access, R read access. We use e-mail addresses for users and developers in ACLs.

Useful groups and services:

@all -- all users with SSH keys on git.g.o,
@devs -- all developers and staffers,
@ldap__repo__gentoo -- all developers with access to main gentoo repository,
daemon -- git:// access,
gitweb -- cgit access/listing.

When the entry is ready, commit and push. Gitolite should print creating new repository ... in status messages.

Replying to the request

After creating the repository, you need to reply to the user with the terms of service. Take docs/template_user_overlay.txt, perform replacements according to the instruction inside, remove the instruction and paste it to the bug.

user $vim docs/template_user_overlay.txt

When user replies that the repository contents have been committed, proceed with adding to repositories.xml.

Removing and renaming repositories

Gitolite doesn't allow us to physically remove repositories from server. Instead, we just disable them (access and visibility). If you need to physically remove the data from a repository, contact Infra.

In order to disable a repository, just comment out its entry from conf/repo.conf. This will remove it from listings and disable all access to it, while preserving the comment as a note that a repository with given path still physically exists.

Moving and renaming repositories is not directly supported. Instead, you create a new repository, disable the old one and ask the user to push the contents again.

Debugging repository access issues

If user is reporting problems accessing a repository, first point them to the access issue debugging guide. If it doesn't help them, it provides useful debugging information.

If gitolite rejects user's SSH key or authenticates him as wrong user, ensure that the files in keydir are correct and have up-to-date SSH keys. Make sure that the keys are not malformed. In particular, there should be no newlines inside the key data (i.e. it should be one long line). If in doubt, get the fingerprint of key file and compare it with the one provided by user:

user $ssh-keygen -l -f keydir/nondevs/foo@example.com@00.pub

2048 SHA256:... foo@example (RSA)

If user is using old OpenSSH version, he may provide MD5 checksum. To get one for comparison, do:

user $ssh-keygen -l -E md5 -f keydir/nondevs/foo@example.com@00.pub

2048 MD5:..:..:... foo@example (RSA)

If user can connect to gitolite yet doesn't have write access to the repository, verify the repository ACL in conf/*.conf. Make sure that the user is there, with correct e-mail address.

If user has used more than one e-mail address in the past, you may need to combine them. In this case:

verify that they all belong to the user somehow,
remove or rename old SSH keys appropriately,
update ACLs to the new e-mail address in all *.conf files.

If user isn't in the ACL, make sure that he should be. If he's the owner, that's fine. If he's not, make sure you get confirmation from the owner. You may look at old bugs if it already hasn't been confirmed there. IRC confirmation is fine too.

GitHub / any external service mirroring

We can optionally enable support for automatic mirroring of the repository on any external hosting like GitHub.

Ensure that the user has read the guide on requesting external hosting, understood the big warning at the top and added our deployment user/key.

Afterwards, mirroring can be enabled using the following line:

FILE conf/repo.conf

repo repo/user/foo
	# ...
	config gentoo.mirror.url = git@github.com:foo/foo-overlay.git