Gitlab Releases and Blobs

When writing code at some point we want to highlight a specific state, we want to celebrate reaching a new milestone and usually this is the moment to prepare a release. Nowadays when we all are about to use git, it is quite common to utilize tags for this. A tag points to a specific commit, it is a persistent shortcut to this point of the development. When using git-flow marking a release is mostly the only use case for tags. The next step is to store a compiled, ready-to-use package – maybe a Docker image – of the software together with a specific tag to bring it all together: version marker and source code, changelog and release notes (within the tag description), and of course binary objects, available via one view that serves as a more verbal and more user friendly history of the software project.

We are using Gitlab that features all what git offers and many, many, many more, that’s for sure. But releases can be created via API only – a web interface to this feature is missing. Even worse that there is no blob (binary large object) store attached to this feature, so the only alternative is to add URLs to the objects.

“No problemo” the little guy from Melmac would say. There are the build artifacts we can store attached to the CI (continues integration) pipeline within Gitlab. But these artifacts have a limited lifetime and will be deleted after a period configured by the Gitlab administration: the admins at our computing center. So we do not have a direct chance to change the configuration here and as of today our artifacts will be preserved for six days only. Even if we specify a longer period within the configuration file (.gitlab-ci.yml) like the following example, the administration will override with a maximum time span: in our case this limit is set to six month.

job:
  artifacts:
    expire_in: 100 years

Again Gitlab offers the option to preserve certain artifacts. This can be triggered manually with a button “Keep” at the job details view or via API.

Lets put this together to get an experience like GitHub users have, featuring releases in this way and automatize this process within a CI pipeline. The Requirement: jq (a JSON parser for the command line) and your favorite HTTP client.

Following the recommendation at the Gitlab documentation, we can set up a job preparing a release and to preserve the artifact. Besides the availability of jq in your container, it requires access rights to the Gitlab API with the help of an access token. Here are some information about. It is RECOMMENDED to mask this variable, as it offers access to many features bound to a single personal account. In addition it can be set to be available only for pipelines triggered form protected branches or protected tags. A secret variable named GITLAB_TOKEN will be used in the following code.

# when a git flow release is made, a tag will be pushed starting this job. it
# will keep the resulting artifact from the job declared in `JOB_NUMBER_TO_PRESERVE`
# and it will set up a Gitlab release at the repo. therefore the merge message starting
# at the release branch should be written in markdown, using a backslash as escape character
# `\` before `#`.

release:
  stage: deploy
  only:
    - tags
  variables:
    JOB_NUMBER_TO_PRESERVE: 1

  script:
    # it is also possible to query for a job name by altering the jq filter
    - 'curl --output jobs.json --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "$CI_API_V4_URL/projects/$CI_PROJECT_ID/pipelines/$CI_PIPELINE_ID/jobs"'
    - CI_JOB_TARGET=$(jq ".[$((JOB_NUMBER_TO_PRESERVE - 1))].id" < jobs.json)
    - echo $CI_JOB_TARGET
    # keep artifact (release will link there)
    - 'curl --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "$CI_API_V4_URL/projects/$CI_PROJECT_ID/jobs/$CI_JOB_TARGET/artifacts/keep"'
    # create release data
    # parse commit message (markdown, lines starting with “\”)
    - MARKDOWN=$(echo "$CI_COMMIT_MESSAGE" | sed 's=^\\==g')
    # prepare the json file
    - 'jq
      ".name = \"$CI_PROJECT_PATH $CI_COMMIT_TAG\" |
      .tag_name = \"$CI_COMMIT_TAG\" |
      .description = \"$MARKDOWN\" |
      .assets.links[0].name = \"package\" |
      .assets.links[0].url = \"https://gitlab.gwdg.de/$CI_PROJECT_PATH/-/jobs/$CI_JOB_TARGET/artifacts/download\"
      " < gitlab-release.json.tmpl > gitlab-release.json'
    - 'curl --header "Content-Type: application/json" --header "PRIVATE-TOKEN: $GITLAB_TOKEN" --data @gitlab-release.json --request POST $CI_API_V4_URL/projects/$CI_PROJECT_ID/releases'
  artifacts:
    paths:
      - gitlab-release.json

This pipeline is mostly parameterized and so easy to adapt (or possibly to be used without any change) in other projects. The job will store the JSON file used to describe the release as a base to start customizing the values with the help of jq.

You can see the results in action:+

• Repo SADE/assets
• Gitlab CI pipeline configuration (gitlab-ci.yml)
• Tag
• Release
• build job details (with keept artifact)
• release job details(with release data as JSON, manually preserved)

We will enter the release notes directly at git flow release finish 'version' and put them into the commit message. Commit Messages are not completely compatible with markdown, even if major web views will render the commit description in accordance. So we have to escape the # for headlines with a leasing backslash. The backslash will be removed for the release description, but unfortunately there is no way to remove it from the commit description.