User:schievel/autocreate rust sources

Schievel

Schievel

Contact info

Webform

deThis user is a native speaker of German.

Automating source and vendor tarball creation

Rust (and Go) programs are using their own package managers to download dependencies before building. The cargo.eclass has some functionality to work use this, using the CRATES variable. Dependencies that are declared in that variable are automatically downloaded and unpacked into the right spot using the cargo.eclass in an ebuild. However, this comes at a cost: Every time the ebuild is updated and upstream uses a different version of the dependency, the CRATES variable needs to be updated, too. Packagers often end up running pycargoebuild (or cargo build) every time there is a new version for that build to get that list of dependencies.

Also upstream repos sometimes make use of git submodules. These submodules are other git repositories that are put into the repository. On GitHub those submodules are not included in the automatically created source tarball of a release. Therefore packagers need to download those repositories separately and put them into the right places in their ebuild manually. (See app-emulation/dxvk's ebuild for details) This includes extra work for packagers again, because we need to get the submodules commit hash for each release and put that in the ebuild. Also the commit hashs are not guaranteed to stay the same forever, so ebuilds using that mechanism could break.

Overview

In the best case we ask upstream to provide proper release tarballs and vendor tarballs with each release. If they are using GitHub actions anyway, this is just a few commands they would need to add to their release action. But sometimes upstream refuses to do that. In this case we are going to set up our own Github Repo that watches upstreams repo for new releases, fetches the upstream repo if there is a new release and packages a source tarball and/or vendor tarball.

Usually Github actions assume that we have write access to that upstream repo and start a Github action from there. We don't have access to upstream's repo, so we need a little workaround for that. We watch upstream repo using a schedule and extract the name of the last tag from it and save that tag. If the last tag differs from the saved tag, we know a new version was released and we save that version. Another github action watches the file (or the folder of that file) for changes and if it changed it starts and creates the tarballs we want.

Caveats

This currently only works when watching another GitHub repo. However, with some little fine tuning by using something different than the actions/checkout@v2 action that is able to check out repositories from other Git forges like GitLab are Bitbucket and by using their API to extract the latest tag from that repo, this could also work with other forges.

Watching another GitHub repo

- create an we repository on Github. - create the file release-versions/latest.txt in that repository. It doesn't matter what is saved in that file, but for starters put 0.0.0 in it. - create the file .github/workflows/watch-release.yaml in the repository. Past the following into that file.

CODE

name: Get latest release version
on:
  schedule:
    - cron:  '20 11,23 * * *'
    
  workflow_dispatch: 
jobs:
  get-version:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
        with:
        token: ${{secrets.REPO_SCOPED_TOKEN }}
      - name: Fetch release version
        run: |
          curl -sL https://api.github.com/repos/upstream-team/upstream-repo-name/releases/latest | \
          jq -r ".tag_name" > release-versions/latest.txt
      - name: Check for modified files
        id: git-check
        run: echo ::set-output name=modified::$([ -z "`git status --porcelain`" ] && echo "false" || echo "true")
      - name: Commit latest release version
        if: steps.git-check.outputs.modified == 'true'
        run: |
          git config --global user.name 'Your name'
          git config --global user.email 'your email'
          git commit -am "New release version"
          git push

This would set up to do that check for a new version twice a day at 11:20 UTC and 23:20 UTC. Set it to different times to your liking Also change upstream-org/upstream-repo-name to the upstream repository you want to watch, e.g. torvalds/linux and lastly put in your name and email. This should now check at the given schedule if there is a new tag upstream and if so, write that tag into the latest.txt file. You can check if that works by clicking on (in your newly created repo) Actions->Get the latest release version (on the left)->Run workflow (on the right)->Run workflow. It should fetch the lastet tag from the upstream repo and write the tags name into release-versions/latest.txt

Creating a Github token with repo scope

Using the above Github actions requires to set up a Github token with sufficient permissions to push to that repo and saving it in the repository secret REPO_SCOPED_TOKEN.

Creating the token

This is thoroughly described in the Github docs but essentially you do: - click on your avatar in the top left, click on settings -> Developer settings -> personal access tokens -> Tokens (classic) -> Generate new token -> Generate new token (classic). Now click yourself through the process and tick the "repos"-permissions. In the end it will give you a token (some long alphanumeric gibberish), copy that.

Setting the token as a repository secret

Again there is the official documentation in Github docs for this. Here is a short explaination what to do: Back in your repo, click on Settings->Secrets and Variables->Actions->New repository secret. As name you set REPO_SCOPED_TOKEN and as secret paste in the alphanumeric gibberish token from the step above.

Using Github Actions to create a release

Now create the file .github/workflows/release.yaml. This workflow will create the release with tarballs from upstream whenever there is a change in the folder release-versions. Put the following into that file:

CODE

name: release

on:
  push:
    paths:
      - 'release-versions/*'
              
  workflow_dispatch:
  
permissions: write-all

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Fetch release version
        run: |
          echo "latest_tag=$(\
          curl -sL https://api.github.com/repos/upstream-team/upstream-repo-name/releases/latest | \
          jq -r ".tag_name")" >> $GITHUB_ENV
          
      - uses: actions/checkout@v3.3.0
        with:
          repository: upstream-team/upstream-repo-name
          submodules: true
          ref: ${{ env.latest_tag }}

      - name: Create tarballs
        run: |
          cd ..
          zip upstream-repo-name-${{ env.latest_tag }}-complete.zip ${{ github.event.repository.name }} -r
          tar -czvf upstream-repo-name-${{ env.latest_tag }}-complete.tar.gz ${{ github.event.repository.name }}
          cd ${{ github.event.repository.name }}
          cargo vendor
          cd ..
          tar -czvf vendor.tar.gz ${{ github.event.repository.name }}/vendor
          
      - uses: "marvinpinto/action-automatic-releases@latest"
        with:
          repo_token: "${{ secrets.GITHUB_TOKEN }}"
          automatic_release_tag: ${{ env.latest_tag }}
          prerelease: true
          files: |
            ../upstream-repo-name-${{ env.latest_tag }}-complete.zip
            ../upstream-repo-name-${{ env.latest_tag }}-complete.tar.gz
            ../vendor.tar.gz

Again you need to put in upstream-org/upstream-repo-name to the upstream repository you want to package. The section under Create tarballs can be changed to whatever you like, e.g. commands to create a Go vendor tarball as well using the methods described here.

Again you can check if that works by running this manually with Actions->release(on the left)->Run workflow (on the right)->Run workflow. This should create a new release in your repo with the tarballs and zip in your repo.

Submodules of submodules

Sadly the checkout in

CODE

- uses: actions/checkout@v3.3.0
        with:
          repository: upstream-team/upstream-repo-name
          submodules: true
          ref: ${{ env.latest_tag }}

will only checkout the direct submodules of that upstream repo. If the submodules have submodules again, it will not check them out. In that special case you could check that submodule repo out separately and put it in the right place. E.g.

CODE

- name: Checkout submodule submodule
        uses: actions/checkout@v3.3.0
        with:
          repository: subsubmodule-org-name/subsubmodule-repo-name
          path: submodule/path/
          ref: 'main'

Using the tarballs in an ebuild

So instead of downloading and putting the submodules into the right places manually in the ebuild, you would just put that tarball you created in the Github actions into SRC_URI. To use a Rust vendor tarball with the cargo.eclass see Tips for packaging Rust.

See also

• Writing_go_Ebuilds — a short reference, intended to be read alongside Basic guide to write Gentoo Ebuilds and the go-module.eclass documentation
• Writing_Rust_ebuilds — a short reference, intended to be read alongside Basic guide to write Gentoo Ebuilds and the cargo.eclass documentation