Project:Infrastructure/Mirrors/Wrangling

From Gentoo Wiki
Jump to: navigation, search

This guide lists the steps needed to properly wrangle new Gentoo mirrors.

Mirror wrangling

Preparing and testing mirrors

1. New mirrors should file a bug under the "Mirrors" product, "New Server" component. If a request comes in via email, ask them to file the request on Bugzilla.

2. Ask for the following information on the bug, if not provided already:

  • Official mirror name (preferably not your hostname, but the name of your organization)
  • Mirror type: distfiles/releases, gentoo-portage. NOTE: gentoo-portage is always rsync only . Some mirrors may have rsync for distfiles as well. Do not be confused by the difference!
  • URLs: full protocol and path for release/distfile mirrors, hostname or IP for gentoo-portage mirrors
  • Hardware specifications
  • Bandwidth specifications (transfer speed, eg 100Mbit, and any limits you may have)
  • Location
  • Intended concurrent users (for each protocol, mirror type)
  • Admin contact name and email address (This address should be registered in bugzilla so we can CC you to any bug reports)

3. Modify the summary to have the type of mirror and the city as well as country if possible.

4. Inform the mirror of testing policies, tell them the due date (when testing will be completed) and give them the following links to make sure their mirror(s) have been setup correctly:

Our policy right now for testing duration is as follows:

  • New distfiles/gentoo-portage mirrors: 2 weeks from date of infra accepting the bug
  • Returning distfiles/gentoo-portage mirrors: Same hardware/IP - 1 week. Different hardware/IP - 2 weeks
  • Problematic mirrors: 1 week to fix the issue, else they're removed from rotation unless we're provided with valid reasons by the admins.
  • Rsync mirrors in any continental rotation on DNS with issues should be removed immediately from those until the issue is resolved. For example, rsync.namerica.gentoo.org , rsync.europe.gentoo.org . You can leave the mirror in its country's rotation.

Now the mirror is ready to be added officially.

Adding mirrors

For distfiles mirrors

1. Add the mirror to GIT git.gentoo.org/data/api/files/mirrors/distfiles.xml (check the format of the doc, follow the same).

{{CodeBox|title=Example email|1=Hi, We're glad to have you as our newest source mirror, welcome aboard!

Use this information to sync with our private mirror every 4 hours from Midnight local time. This is for mirror admins only, please do not share this information.


rsync module: masterdistfiles-{eu,us}.gentoo.org::gentoo

rsync -rlptDq --delete -H \ masterdistfiles-{eu,us}..gentoo.org::gentoo /your/local/path/goes/here

Please note that we have received several reports of problems when using the -z (compress) option of rsync. Thus, please DO NOT use this option when syncing against this mirror.


Please also make sure you are subscribed to gentoo-mirrors@gentoo.org as we often send out administrative notices and policy changes via this list.

Thanks for helping Gentoo!

Cheers, $name }}

3. Add the mirror/admin-email tothe infra wiki .

4. Put in the following comment and close the bug:

Thanks for helping out Gentoo. I have added your mirror to our list at http://www.gentoo.org/main/en/mirrors2.xml. Please start syncing from masterdistfiles.gentoo.org as the email I have sent to you instructs. We would like you to subscribe to the gentoo-mirrors Mailing List by sending an email to gentoo-mirrors+subscribe@lists.gentoo.org, as we send out notifications of updates, releases, and changes in policy to the mirrors. Please also monitor http://mirrorstats.gentoo.org as this application is used to keep track of the up-to-date nature of our mirrors. Thanks for the hardware and bandwidth!

For gentoo-portage mirrors

1. Add new host to the ACLs . Follow the format, re-use existing numbers for a country before adding new ones.

CODE Adding an IP to the ACL
$ git checkout git+ssh://git@git.gentoo.org/infra/masterportage-acl
$ ${EDITOR} hosts.allow.master
$ git commit hosts.allow.master

2. To add a new country, just make sure it isn't already there, then create one.

Note
If you've added a new country to this file, chances are countryTLD.gentoo.org doesn't exist, check FIRST and if nothing, create one using the "Add A Domain Page". Email infra@gentoo.org(CC mirror-admin@gentoo.org) to notify them of the new sub-domain.

3. Add the IP into the country (and if needed, continent) DNS rotation(s) in git-based DNS. For example, a new mirror in the US would be rsync(number).us.gentoo.org , so you will add the new IP:

CODE Managing DNS rotation
rsync(number).us.gentoo.org - TTL - 86400

(US/Country rotation)
rsync.us.gentoo.org - TTL - 1800

(Continental rotation)
rsync.namerica.gentoo.org - TTL - 1800
Note
TTLs for rotations are obviously lower, so that we can modify them quickly if needed.

4. Add the mirror to git git.gentoo.org/data/api/files/mirrors/rsync.xml (check the format of the doc, follow the same).

5. Add the mirror/admin-email tothe infra wiki .

6. Put in the following comment (modify as needed) and close the bug:

Thanks for helping out Gentoo. I have added the IP you specified in the bug to our access database, please start syncing to masterportage.gentoo.org per our rsync document at http://www.gentoo.org/doc/en/rsync.xml. We would like you to subscribe to the gentoo-mirrors Mailing List by sending an email to gentoo-mirrors+subscribe@lists.gentoo.org, as we send our notifications of updates, releases, and changes in policy to the mirrors. Once again, thanks for the hardware and bandwidth! Please also monitor http://mirrorstats.gentoo.org/rsync as this application is used to keep track of the up-to-date nature of our mirrors. You are mirror rsync(number).$country.gentoo.org in the rsync.$country.gentoo.org rotation and have been added to the rsync.$continent.gentoo.org rotation as well.

Removing mirrors

For distfiles mirrors

1. Remove the mirror from git git.gentoo.org/data/api/files/mirrors/distfiles.xml (check the format of the doc, follow the same).

2. Remove the mirror/admin-email fromthe infra wiki .

3. Put in the following comment and close the bug:

Removed $server_name from mirrors listings. Thanks for the hardware and bandwidth donation, your services helped many Gentoo Users. (You have no more obligation to monitor the gentoo-mirrors mailing list.)  

For gentoo-portage mirrors

1. Remove the IP from osprey.gentoo.org , at /home/gmirror/rsync-hosts-allow/hosts.allow.master . Leave the existing rsyncX.$CC comment so that it is known that it should be reused.

CODE Removing an IP from the ACL
$ git checkout git+ssh://git@git.gentoo.org/infra/masterportage-acl
$ ${EDITOR} hosts.allow.master
$ git commit hosts.allow.master
Note
If you are removing the last mirror in a rsync.$CC rotation, you should talk to infra and make sure there is a solution so users do not try to sync to an dead address.

2. Remove the IP from the country DNS rotation(s) in git-based DNS

3. Remove the mirror from CVS gentoo/xml/htdocs/main/en/mirrors-rsync-data.xml (check the format of the doc, follow the same).

4. Put in the following comment (modify as needed) and close the bug:

Removed $server_name from ACLs, removed from DNS. Thanks for the hardware and bandwidth donation, your services helped many Gentoo Users. (You have no more obligation to monitor the gentoo-mirrors mailing list.)  

This page is based on a document formerly found on our main website gentoo.org.
The following people contributed to the original document: robbat2, fox2mike, nightmorph
They are listed here because wiki history does not allow for any external attribution. If you edit the wiki article, please do not add yourself here; your contributions are recorded on each article's associated history page.