Building Docker images for multiple processor architectures

NOTE: This article is has a continuation.

I’ve been a happy user of Intel platform since I saw an IBM PC/XT in my university lab. But I’m afraid that the times are changing. ARM platform, which is literally in all the mobile devices and tablets, is now finding its way to cloud servers and notebooks.

At the end of the last year, I bought three ROCK64 SBC and build a pico cluster for my experiments with Kubernetes. Everything looks good and solid but the reality is that now I have to develop cross-platform applications. Obviously, I want to develop them using amd64 based platforms (Windows or MacOS, at least for now) and run them on my ROCK64s with arm64/v8 architecture.

Luckily enough, it is quite easy to make cross-platform applications in Golang but publishing platform-dependant images in a Docker container registry and using manifests is a quite convoluted process. This article summarizes my experience in such process automation on Travis CI.

Preparing for the build

1. First of all, you need to use Ubuntu xenial which is not default on Travis CI currently, plus sudo is required for some commands, therefore in you .travis.yml you need to specify:

sudo: required

dist: xenial

2. You also require Docker service and update it to the latest version:

services:
  - docker

addons:
  apt:
    packages:
      - docker-ce

3. Using environment variables you need to enable experimental Docker client features, since the manifest command is an experimental one, and set BUILDKIT_HOST for the buildkit client – buildctl:

env:
  global:
    - BUILDKIT_HOST=tcp://0.0.0.0:1234
    - DOCKER_CLI_EXPERIMENTAL=enabled

4. During the setup phase, since experimental –platform option is used later, you need to enable experimental mode for Docker daemon too:

echo '{"experimental":true}' | sudo tee /etc/docker/daemon.json
sudo service docker restart

5. Register file format recognizers for different architectures by running binfmt container in privileged mode:

sudo docker run --privileged linuxkit/binfmt:v0.6

6. The build is going to use buildkit and we need to start the server part of it.

sudo docker run -d --privileged \
  -p 1234:1234 \
  --name buildkit moby/buildkit:latest \
  --addr tcp://0.0.0.0:1234 \
  --oci-worker-platform linux/amd64 \
  --oci-worker-platform linux/arm64/v8

The actual script (setup.sh) is using some variables to achieve better flexibility but the idea is the same.

7. At the last step, we are extracting buildkit’s command line utility – buildctl from the container and put it to /usr/bin so that we can run it later.

sudo docker cp buildkit:/usr/bin/buildctl /usr/bin/

Building the images

The image building (build.sh) is quite trivial assuming that you are using a multi-platform base image such as golang:alpine. In my case, because I’m building a build image for my other containerized applications, I just need to build dep tool using Dockerfile and the resulting image is going to be platform specific automagically, given that we specify platform option correctly.  I’m not going to go into the details of the Dockerfile itself but for building an image using buildkit I’m using this construct:

mkfifo fifo.tar
trap 'rm fifo.tar' EXIT

buildctl build --frontend dockerfile.v0 \
  --frontend-opt platform="${platform}" \
  --local dockerfile=. \
  --local context=. \
  --exporter docker \
  --exporter-opt name="${image}:${platform_tag}" \
  --exporter-opt output=fifo.tar \
  & docker load < fifo.tar & wait

I’m not pushing the image I build immediately from buildkit to any container registry. Instead, I’m feeding it back to the local Docker service using a named pipe because I might want to test it before pushing (currently not implemented).

The command could’ve been much simpler but I found that you can’t use a normal Linux pipe (|) because the progress output from buildkit is mixed with its stdout (an issue submitted).

Pushing the images and manifests

If the image building is quite straight forward the next steps are heavily dependent on your publishing strategy. I decided to implement the following algorithm (see details in push.sh):

1. As it was mentioned before, the build produces platform-specific images and tags them using the target platform name and a part of the current commit tag (e.g. foo/bar:12be34-linux-amd64 or foo/bar:12be34-linux-arm64-v8) then it pushes images to the local Docker cache so that they can be run for testing.
2. When a Git commit is tagged, hopefully with a tag using semantic versioning format (e.g. v1.2.3), the deployment process is triggered. During the deployment, all the platform-specific images built at step #1 are pushed to the Docker registry:

docker push foo/bar:12be34-linux-amd64
docker push foo/bar:12be34-linux-arm64-v8

3. Docker multi-platform manifests are created using the semantic version provided in a tag. For example, if tag v1.2.3 was used for a commit, multi-platform manifests are created for tags foo/bar:1, foo/bar:1.2, foo/bar:1.2.3, foo/bar:latest and all of them have a list of tags pointing to platform-specific images already pushed to Docker Hub:

docker manifest create --amend foo/bar:1 foo/bar:12be34-linux-amd64 foo/bar:12be34-linux-arm64-v8
docker manifest create --amend foo/bar:1.2 foo/bar:12be34-linux-amd64 foo/bar:12be34-linux-arm64-v8
docker manifest create --amend foo/bar:1.2.3 foo/bar:12be34-linux-amd64 foo/bar:12be34-linux-arm64-v8
docker manifest create --amend foo/bar:latest foo/bar:12be34-linux-amd64 foo/bar:12be34-linux-arm64-v8

5. All the created manifests are pushed to Docker Hub in order of tags: 1, 1.2, 1.2.3, latest. The order is important if you want to get the more recent tags at the top of the tags list. For example:

docker manifest push --purge foo/bar:1
docker manifest push --purge foo/bar:1.2
docker manifest push --purge foo/bar:1.2.3
docker manifest push --purge foo/bar:latest

Cleanup

Now we have all the semantic tags and platform-specific tags pointing to the images we pushed earlier.

But you don’t need the platform-specific tags (green) anymore and you can safely delete them from the Docker registry. Your platform-specific images (orange) are still going to stay because they are referenced by the tags with semantic versions (blue) and the latest tag.

Unfortunately, there is no Docker command for doing that, so you have no choice but to use RESTful API and send DELETE using bearer token authorization:

local token=$(curl -s -X POST \
  -H "Content-Type: application/json" \
  -d '{"username": "'"$DOCKER_USERNAME"'", "password": "'"$DOCKER_PASSWORD"'"}' \
  https://hub.docker.com/v2/users/login/ | jq -r .token)

local code=$(curl -s -o /dev/null -LI -w "%{http_code}" \
  https://hub.docker.com/v2/repositories/"${image}"/tags/"${tag}"/ \
  -X DELETE \
  -H "Authorization: JWT ${token}")

Pushing REAMDE.md

If you ever configured builds on Docker Hub you probably have noticed that your README.md file is fetched when a build succeeds and displayed on your repository page. It is very convenient but when you build outside of Docker Hub you have to push it by yourself. Fortunately, Docker Hub has a nice API not only for pushing images but also for doing some tricks with your repository. Using this API you can push a new version of REARME.md every time your build pushes images.

local code=$(jq -n --arg msg "$(<README.md)" \
    '{"registry":"registry-1.docker.io","full_description": $msg }' | \
        curl -s -o /dev/null  -L -w "%{http_code}" \
           https://cloud.docker.com/v2/repositories/"${image}"/ \
           -d @- -X PATCH \
           -H "Content-Type: application/json" \
           -H "Authorization: JWT ${token}")

The same token mentioned in the previous topic can be used for this call.

Notes

Adding additional platform

The scripts I tried to combine for my golang-dep repository are quite flexible and in order to add an additional platform you just need to extend the comma-separated list of supported platforms. I’m currently building for linux/amd64, linux/arm/v6, and linux/arm64/v8.

- DOCKER_PLATFORMS="linux/amd64,linux/arm/v6,linux/arm64/v8"

Using multi-platform images

Normally Docker should recognize the platform you are using and fetch platform-specific images accordingly. But if you want to specify a platform explicitly you can always do that using experimental –platform option of docker run and docker pull.

Removing locally cached manifests

If you, for some reason, decide to delete a locally cached manifest you won’t find any docker command for that. But there is a way still, you just delete the manifest’s directory in ~/.docker/manifests/ You might need to do that because –amend is not working properly and manifests are not purged if manifest push fails for some reason.

Why not Golang 1.11.0/1.12.0?

When I tried to switch my Dockerfile to use golang:1.12-alpine I discovered that I can’t build anymore for linux/arm64/v8 architecture using buildkit. It was giving me this error:

error: failed to solve: rpc error: code = Unknown desc = failed to copy: httpReaderSeeker: failed open: 
could not fetch content descriptor sha256:7cf1f7ccf392bd834eb91f02892f48992d3c2ba292c2198315a4637bb9454c30 
(application/vnd.docker.distribution.manifest.v1+json) from remote: not found

The issues were reported to golang and buildkit.

Manifests annotation

Currently produced manifests are not annotated with architecture variants. So if you inspect manifests using weshigbee/manifest-tool:

docker run --rm weshigbee/manifest-tool inspect moikot/golang-dep

You will see no v6 or v8 specified:

...
1    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
1       Digest: sha256:01d07e1cb4b45480965e2c1595345a7c5e1b786d22d67b20287998607ffeeff4
1  Mfst Length: 1575
1     Platform:
1           -      OS: linux
1           -    Arch: arm
1           - Variant: 
1           - Feature: 
1     # Layers: 6
...
2    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
2       Digest: sha256:e244de34eae88283266b70e4679e79db482ae8e2154a2ce354b1691e3f2db3e0
2  Mfst Length: 1576
2     Platform:
2           -      OS: linux
2           -    Arch: arm64
2           - Variant: 
2           - Feature: 
2     # Layers: 6
...

This will be fixed soon, stay tuned!