How to set multi arch annotations if different platforms are built on different runners?

[TKrisee]

I'm trying to add annotations to our docker images while building for multiple platforms. The relevant part of the dockerfile is as follows:

FROM ubuntu:22.04 as app

LABEL maintainer="Inline Studio"
LABEL org.opencontainers.image.description "Base Docker image for Laravel development"

and the workflows looks like this:

name: ci

on:
  workflow_dispatch:
  push:
    branches:
      - main
      
env:
  REGISTRY: ghcr.io

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  build:
    strategy:
      fail-fast: false
      matrix:
        platform:
          - linux/amd64
          - linux/arm64
        image:
          - app
          - database
    permissions:
      contents: read
      packages: write
    runs-on: ${{ matrix.platform == 'linux/amd64' && 'ubuntu-latest' || fromJSON('["self-hosted", "ARM64"]') }}
    name: Build Docker image ${{ matrix.image }} ${{ matrix.platform }}
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      - name: Docker meta
        id: meta
        uses: docker/metadata-action@v5
        with: 
          images: ${{ env.REGISTRY }}/${{ github.repository }}-${{ matrix.image }}
      - name: Set up Docker Context for Buildx
        id: buildx-context
        run: |
          docker context create builders
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
        with:
          endpoint: builders
          platforms: ${{ matrix.platform }}
      - name: Login to GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Build and push by digest
        id: build
        uses: docker/build-push-action@v5
        with:
          provenance: false
          context: .
          target: ${{ matrix.image }}
          platforms: ${{ matrix.platform }}
          labels: ${{ steps.meta.outputs.labels }}
          annotations: ${{ steps.meta.outputs.annotations }}
          outputs: |
            type=image,name=${{ env.REGISTRY }}/${{ github.repository }}-${{ matrix.image }},push-by-digest=true,name-canonical=true,push=true
          cache-from: type=gha,scope=${{ github.repository }}-${{ github.ref_name }}-${{ matrix.platform }}-${{ matrix.image }}
          cache-to: type=gha,scope=${{ github.repository }}-${{ github.ref_name }}-${{ matrix.platform }}-${{ matrix.image }}
          secrets: |
            "COMPOSER_AUTH=${{ secrets.COMPOSER_AUTH }}"
      - name: Export digest
        run: |
          mkdir -p /tmp/digests/${{ matrix.image }}
          digest="${{ steps.build.outputs.digest }}"
          touch "/tmp/digests/${{ matrix.image }}/${digest#sha256:}"
      - name: Upload digest
        uses: actions/upload-artifact@v3
        with:
          name: ${{ matrix.image }}-digests
          path: /tmp/digests/${{ matrix.image }}/*
          if-no-files-found: error
          retention-days: 1
      - name: Clear digest
        run: |
          rm -rf /tmp/digests/${{ matrix.image }}

  merge:
    strategy:
      fail-fast: false
      matrix:
        image:
          - app
          - database
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    needs:
      - build
    name: Merge Docker manifests
    steps:
      - name: Download digests
        uses: actions/download-artifact@v3
        with:
          name: ${{ matrix.image }}-digests
          path: /tmp/digests/${{ matrix.image }}
      - name: Docker meta
        id: meta
        uses: docker/metadata-action@v5
        with: 
          images: ${{ env.REGISTRY }}/${{ github.repository }}-${{ matrix.image }}
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      - name: Login to GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Create manifest list and push
        working-directory: /tmp/digests/${{ matrix.image }}
        run: |
          docker buildx imagetools create $(jq -cr '.tags | map("-t " + .) | join(" ")' <<< '${{ steps.meta.outputs.json }}') \
            $(printf '${{ env.REGISTRY }}/${{ github.repository }}-${{ matrix.image }}@sha256:%s ' *)
      - name: Inspect image
        run: |
          docker buildx imagetools inspect '${{ env.REGISTRY }}/${{ github.repository }}-${{ matrix.image }}:${{ steps.meta.outputs.version }}'

the ARM images are built on raspberry pi-s using github ARC, while the x86 ones are built on the default github runners. My problem is that once I check the images on GHCR they have no description available:

[oklopfer]

Finally figured it out. Buildx version is currently outdated (v0.11.2 vs v0.12.0). Issue is solved by adding:

      - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v3
         with:
++         version: v0.12.0
      [...]
      - name: Create manifest list and push
         working-directory: /tmp/digests
         run: |
           docker buildx imagetools create $(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
++           --annotation "index:org.opencontainers.image.description=Your description goes here" \
             $(printf '${{ env.REGISTRY_IMAGE }}@sha256:%s ' *)

However I say "solved" because it adds significant time to the build in doing the update to v0.12.0.
EDIT: seems to have only stalled the first time. Works just fine now with the pin :D

[NilashishC]

I seem to have the same problem with docker buildx version v0.13.0 0de5f1c. This is the command that I'm using:

docker buildx imagetools create -t ghcr.io/ansible/community-ansible-dev-tools-container:test \
 --annotation "index:org.opencontainers.image.description=A multi arch community container image for Ansible Devtools." \
 ghcr.io/ansible/community-ansible-dev-tools-container:latest@sha256:fbe77ed1c5dff9a5012bc4bfd879bedea07a22ce5af9222a35a4b0cdd8e13f2d ghcr.io/ansible/community-ansible-dev-tools-container:latest@sha256:e71820b7fe2692900f56dd6112d6f93120a437bfacd1708a2b060d19c94cd97c

I've followed the same steps as OP. Push the arm64/x86 images separately by digest and then create a manifest from those. The individual images have the annotations (since it's in the container file), but the final manifest doesn't, even after setting it with --annotation. Any clue as to what might be happening here?

https://github.com/ansible/community-ansible-dev-tools-container/pkgs/container/community-ansible-dev-tools-container/192160238?tag=test

[y-nk]

i was reading this https://docs.docker.com/build/exporters/image-registry/#annotations and thought that maybe the --annotation can be replaced by a line in outputs?

[steadyfall]

Doesn't work. The image build fails with the following error: index annotations not supported for single platform export

[steadyfall]

Faced a similar issue and spent a day on this to find something that actually worked --

adding oci-mediatypes=true to .jobs.build.steps[] | select(.id == "build") | .with.outputs (with respect to the workflow OP provided)

that is, set step.with.outputs to oci-mediatypes=true in the docker/build-push-action Github Action.

[MaximeWewer]

Hi,

I had the same problem and after some research this works for me, example here : https://github.com/MaximeWewer/OpenLDAP_prometheus_exporter/blob/d75caa74209cff1eb3b8d70aba8fa53b3058d274/.github/workflows/build-and-release.yml#L121

GitHub Container Registry (GHCR) supports both OCI and Docker v2 Schema 2 formats.
However, to ensure that annotations (such as description) are preserved at the multi-architecture manifest level, images must be pushed in OCI format.

Setting oci-mediatypes=true ensures that the images are built and pushed using OCI-compliant media types. This allows proper annotation of the manifest index later using docker buildx imagetools create.

I hope this helps you ^^