User:Schievel/autocreate rust sources

= Automating source and vendor tarball creation = Rust (and Go) programs are using their own package managers to download dependencies before building. The cargo.eclass has some functionality to work use this, using the CRATES variable. Dependencies that are declared in that variable are automatically downloaded and unpacked into the right spot using the cargo.eclass in an ebuild. However, this comes at a cost: Every time the ebuild is updated and upstream uses a different version of the dependency, the CRATES variable needs to be updated, too. Packagers often end up running  (or  ) every time there is a new version for that build to get that list of dependencies.

Also upstream repos sometimes make use of git submodules. These submodules are other git repositories that are put into the repository. On GitHub those submodules are not included in the automatically created source tarball of a release. Therefore packagers need to download those repositories separately and put them into the right places in their ebuild manually. (See 's ebuild for details) This includes extra work for packagers again, because we need to get the submodules commit hash for each release and put that in the ebuild. Also the commit hashs are not guaranteed to stay the same forever, so ebuilds using that mechanism could break.

= Overview = In the best case we ask upstream to provide proper release tarballs and vendor tarballs with each release. If they are using GitHub actions anyway, this is just a few commands they would need to add to their release action. But sometimes upstream refuses to do that. In this case we are going to set up our own Github Repo that watches upstreams repo for new releases, fetches the upstream repo if there is a new release and packages a source tarball and/or vendor tarball.

Usually Github actions assume that we have write access to that upstream repo and start a Github action from there. We don't have access to upstream's repo, so we need a little workaround for that. We watch upstream repo using a schedule and extract the name of the last tag from it and save that tag. If the last tag differs from the saved tag, we know a new version was released and we save that version. Another github action watches the file (or the folder of that file) for changes and if it changed it starts and creates the tarballs we want.

Caveats
This currently only works when watching another GitHub repo. However, with some little fine tuning by using something different than the  action that is able to check out repositories from other Git forges like GitLab are Bitbucket and by using their API to extract the latest tag from that repo, this could also work with other forges.

Watching another GitHub repo
- create an we repository on Github. - create the file  in that repository. It doesn't matter what is saved in that file, but for starters put  in it. - create the file  in the repository. Past the following into that file.

This would set up to do that check for a new version twice a day at 11:20 UTC and 23:20 UTC. Set it to different times to your liking Also change  to the upstream repository you want to watch, e.g.   and lastly put in your name and email. This should now check at the given schedule if there is a new tag upstream and if so, write that tag into the latest.txt file. You can check if that works by clicking on (in your newly created repo) Actions->Get the latest release version (on the left)->Run workflow (on the right)->Run workflow. It should fetch the lastet tag from the upstream repo and write the tags name into

Creating a Github token with repo scope
Using the above Github actions requires to set up a Github token with sufficient permissions to push to that repo and saving it in the repository secret REPO_SCOPED_TOKEN.

Creating the token
This is thoroughly described in the Github docs but essentially you do: - click on your avatar in the top left, click on settings -> Developer settings -> personal access tokens -> Tokens (classic) -> Generate new token -> Generate new token (classic). Now click yourself through the process and tick the "repos"-permissions. In the end it will give you a token (some long alphanumeric gibberish), copy that.

Setting the token as a repository secret
Again there is the official documentation in Github docs for this. Here is a short explaination what to do: Back in your repo, click on Settings->Secrets and Variables->Actions->New repository secret. As name you set REPO_SCOPED_TOKEN and as secret paste in the alphanumeric gibberish token from the step above.

Using Github Actions to create a release
Now create the file. This workflow will create the release with tarballs from upstream whenever there is a change in the folder. Put the following into that file:

Again you need to put in  to the upstream repository you want to package. The section under  can be changed to whatever you like, e.g. commands to create a Go vendor tarball as well using the methods described here.

Again you can check if that works by running this manually with Actions->release(on the left)->Run workflow (on the right)->Run workflow. This should create a new release in your repo with the tarballs and zip in your repo.

Submodules of submodules
Sadly the checkout in

will only checkout the direct submodules of that upstream repo. If the submodules have submodules again, it will not check them out. In that special case you could check that submodule repo out separately and put it in the right place. E.g.

= See also =



Using the tarballs in an ebuild
So instead of downloading and putting the submodules into the right places manually in the ebuild, you would just put that tarball you created in the Github actions into SRC_URI. To use a Rust vendor tarball with the cargo.eclass see Tips for packaging Rust.