Question 84211 in Docker Question posted in
The official documentation can be found here.

How to download docker image using HTTP API using docker hub credentials

So I have a private repository at docker hub
and I am trying to download image (blobs) manually using HTTP API.

Now there are some issues

But there is no API in docker HUB api to get list of blobs from a tag and then download it.

There is a docker registry api, but there my username password does not work.
What to do?

Tags:

2

Answers


  1. 0

    For an image, you first pull the manifest and parse that for the list of blobs that you need to pull. All of these API’s need the same authorization headers used to list the tags. I’m going to be using regctl from my regclient project to query a local registry, but you could also use curl against Hub which I show below.

    $ regctl tag ls localhost:5000/library/alpine
3
3-bkup-20210904
3.10
3.11
3.12
3.13
3.14
3.2
3.3
3.4
3.5
3.6
3.7
3.8
3.9
latest

$ regctl manifest get localhost:5000/library/alpine:latest --format body | jq .
{
  "manifests": [
    {
      "digest": "sha256:14b55f5bb845c7b810283290ce057f175de87838be56f49060e941580032c60c",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      },
      "size": 528
    },
    {
      "digest": "sha256:40f396779ba29da16f29f780963bd4ad5b7719e3eb5dec04516d583713256aa8",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "arm",
        "os": "linux",
        "variant": "v6"
      },
      "size": 528
    },
    {
      "digest": "sha256:392d9d85dff31e34d756be33579f05ef493cb1b0edccc36a11b3295365553bfd",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "arm",
        "os": "linux",
        "variant": "v7"
      },
      "size": 528
    },
    {
      "digest": "sha256:4fb53f12d2ec18199f16d7c305a12c54cda68cc622484bfc3b7346a44d5024ac",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "arm64",
        "os": "linux",
        "variant": "v8"
      },
      "size": 528
    },
    {
      "digest": "sha256:e8d9cf28250078f08e890a3466efbefda68a8feac03cc4076d3ada3397370d6e",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "386",
        "os": "linux"
      },
      "size": 528
    },
    {
      "digest": "sha256:d860569a59af627dafee0b0f2b8069e31b07fbdaebe552904dbaec28047ccf64",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "ppc64le",
        "os": "linux"
      },
      "size": 528
    },
    {
      "digest": "sha256:6640b198347e5bf1e9a9dc5fc864e927154275dc31f3d26193b74350a5c94c9f",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "s390x",
        "os": "linux"
      },
      "size": 528
    }
  ],
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "schemaVersion": 2
}

$ regctl manifest get localhost:5000/library/alpine@sha256:14b55f5bb845c7b810283290ce057f175de87838be56f49060e941580032c60c --format body | jq .
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 1472,
    "digest": "sha256:e9adb5357e84d853cc3eb08cd4d3f9bd6cebdb8a67f0415cc884be7b0202416d"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 2812636,
      "digest": "sha256:3d243047344378e9b7136d552d48feb7ea8b6fe14ce0990e0cc011d5e369626a"
    }
  ]
}

$ regctl blob get localhost:5000/library/alpine sha256:e9adb5357e84d853cc3eb08cd4d3f9bd6cebdb8a67f0415cc884be7b0202416d | jq .
{
  "architecture": "amd64",
  "config": {
    "Hostname": "",
    "Domainname": "",
    "User": "",
    "AttachStdin": false,
    "AttachStdout": false,
    "AttachStderr": false,
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ],
    "Cmd": [
      "/bin/sh"
    ],
    "Image": "sha256:e211ac20c5c7aaa4ed30d5553654d4679082ec48efcb4d164bac6d50d62653fd",
    "Volumes": null,
    "WorkingDir": "",
    "Entrypoint": null,
    "OnBuild": null,
    "Labels": null
  },
  "container": "b6ba94212561a8075e1d324fb050db160e25035ffcfbbe5b410e411e2b7000e2",
  "container_config": {
    "Hostname": "b6ba94212561",
    "Domainname": "",
    "User": "",
    "AttachStdin": false,
    "AttachStdout": false,
    "AttachStderr": false,
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ],
    "Cmd": [
      "/bin/sh",
      "-c",
      "#(nop) ",
      "CMD ["/bin/sh"]"
    ],
    "Image": "sha256:e211ac20c5c7aaa4ed30d5553654d4679082ec48efcb4d164bac6d50d62653fd",
    "Volumes": null,
    "WorkingDir": "",
    "Entrypoint": null,
    "OnBuild": null,
    "Labels": {}
  },
  "created": "2022-03-17T04:01:59.188838147Z",
  "docker_version": "20.10.12",
  "history": [
    {
      "created": "2022-03-17T04:01:58.883733237Z",
      "created_by": "/bin/sh -c #(nop) ADD file:cf4b631a115c2bbfbd81cad2d3041bceb64a8136aac92ba8a63b6c51d60af764 in / "
    },
    {
      "created": "2022-03-17T04:01:59.188838147Z",
      "created_by": "/bin/sh -c #(nop)  CMD ["/bin/sh"]",
      "empty_layer": true
    }
  ],
  "os": "linux",
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:5e03d8cae8773cb694fff1d55da34a40d23c2349087ed15ce68476395d33753c"
    ]
  }
}

$ regctl blob get localhost:5000/library/alpine sha256:3d243047344378e9b7136d552d48feb7ea8b6fe14ce0990e0cc011d5e369626a | tar -tvzf - | head
drwxr-xr-x 0/0               0 2022-03-16 16:15 bin/
lrwxrwxrwx 0/0               0 2022-03-16 16:15 bin/arch -> /bin/busybox
lrwxrwxrwx 0/0               0 2022-03-16 16:15 bin/ash -> /bin/busybox
lrwxrwxrwx 0/0               0 2022-03-16 16:15 bin/base64 -> /bin/busybox
lrwxrwxrwx 0/0               0 2022-03-16 16:15 bin/bbconfig -> /bin/busybox
-rwxr-xr-x 0/0          824984 2022-02-02 13:21 bin/busybox
lrwxrwxrwx 0/0               0 2022-03-16 16:15 bin/cat -> /bin/busybox
lrwxrwxrwx 0/0               0 2022-03-16 16:15 bin/chgrp -> /bin/busybox
lrwxrwxrwx 0/0               0 2022-03-16 16:15 bin/chmod -> /bin/busybox
lrwxrwxrwx 0/0               0 2022-03-16 16:15 bin/chown -> /bin/busybox
...

    A few examples if you try doing this with curl:

    Get a token (specific to Docker Hub, each registry may have different auth methods and servers):

    token=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" 
        | jq -r '.token')

    To use a user/password rather than an anonymous login, call curl with the -u user:pass option:

    token=$(curl -u "$user:$pass" -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" 
        | jq -r '.token')

    Tags:

    curl -H "Authorization: Bearer $token" 
     -s "https://registry-1.docker.io/v2/${repo}/tags/list" | jq .

    Manifests:

    api="application/vnd.docker.distribution.manifest.v2+json"
apil="application/vnd.docker.distribution.manifest.list.v2+json"
curl -H "Accept: ${api}" -H "Accept: ${apil}" 
     -H "Authorization: Bearer $token" 
     -s "https://registry-1.docker.io/v2/${repo}/manifests/${sha:-$tag}" | jq .

    And blobs:

    curl -H "Authorization: Bearer $token" 
     -s -L -o - "https://registry-1.docker.io/v2/${repo}/blobs/${digest}"

    If you want to export everything to be able to later import, regctl image export will convert the image to a tar of manifests and blobs. This is output in the OCI Layout format, and if you only pull a single image manifest (and not a multi-platform manifest) it will also include the needed files for docker load to import the tar.

    Login or Signup to reply.
  2. 0

    There is a docker registry api, but there my username password does not work. What to do?

    It probably depends on how you supply them. I didn’t try it with a private repository on Docker Hub, but here’s a script that downloads public images from Docker Hub, and images from private registries:

    #!/usr/bin/env bash
set -eu
image=$1
creds=${2-}

# https://github.com/moby/moby/blob/v20.10.18/vendor/github.com/docker/distribution/reference/normalize.go#L29-L57
# https://github.com/moby/moby/blob/v20.10.18/vendor/github.com/docker/distribution/reference/normalize.go#L88-L105
registry=${image%%/*}
if [ "$registry" = "$image" ] 
|| { [ "`expr index "$registry" .:`" = 0 ] && [ "$registry" != localhost ]; }; then
    registry=docker.io
else
    image=${image#*/}
fi
if [ "$registry" = docker.io ] && [ "`expr index "$image" /`" = 0 ]; then
    image=library/$image
fi
if [ "`expr index "$image" :`" = 0 ]; then
    tag=latest
else
    tag=${image#*:}
    image=${image%:*}
fi
if [ "$registry" = docker.io ]; then
    registry=https://registry-1.docker.io
elif ! [[ "$registry" =~ ^localhost(:[0-9]+)$ ]]; then
    registry=https://$registry
fi

r=`curl -sS "$registry/v2/" 
    -o /dev/null 
    -w '%{http_code}:%header{www-authenticate}'`
http_code=`echo "$r" | cut -d: -f1`
curl_args=(-sS -H 'Accept: application/vnd.docker.distribution.manifest.v2+json')
if [ "$http_code" = 401 ]; then
    if [ "$registry" = https://registry-1.docker.io ]; then
        header_www_authenticate=`echo "$r" | cut -d: -f2-`
        header_www_authenticate=`echo "$header_www_authenticate" | sed -E 's/^Bearer +//'`
        split_into_lines() {
            sed -Ee :1 -e 's/^(([^",]|"([^"]|")*")*),/1n/; t1'
        }
        header_www_authenticate=`echo "$header_www_authenticate" | split_into_lines`
        extract_value() {
            sed -E 's/^[^=]+="(([^"]|")*)"$/1/; s/\(.)/1/g'
        }
        realm=$(echo "$header_www_authenticate" | grep '^realm=' | extract_value)
        service=$(echo "$header_www_authenticate" | grep '^service=' | extract_value)
        scope=repository:$image:pull
        token=`curl -sS "$realm?service=$service&scope=$scope" | jq -r .token`
        curl_args+=(-H "Authorization: Bearer $token")
    else
        curl_args+=(-u "$creds")
    fi
fi
manifest=`curl "${curl_args[@]}" "$registry/v2/$image/manifests/$tag"`
config_digest=`echo "$manifest" | jq -r .config.digest`
config=`curl "${curl_args[@]}" -L "$registry/v2/$image/blobs/$config_digest"`
layers=`echo "$manifest" | jq -r '.layers[] | .digest'`
echo "$layers" | 
    while IFS= read -r digest; do
        curl "${curl_args[@]}" -L "$registry/v2/$image/blobs/$digest" | wc -c
    done

    Usage:

    $ ./download-image.sh hello-world
$ ./download-image.sh library/hello-world
$ ./download-image.sh docker.io/library/hello-world
$ ./download-image.sh myregistry.com/hello-world testuser:testpassword
$ ./download-image.sh localhost:5000/hello-world

    Or to be more precise it retrieves the image config and the layers. The thing is, what are you going to do with them? You can try to create a tar archive that looks like the one produced by docker save. Which is what docker-drag basically does. But ideally for that you should know what exactly docker pull and docker save do.

    To give you some links to the source code, docker pull (the server part) more or less starts here:

    https://github.com/moby/moby/blob/v20.10.18/api/server/router/image/image.go#L37
    https://github.com/moby/moby/blob/v20.10.18/api/server/router/image/image_routes.go#L78
    https://github.com/moby/moby/blob/v20.10.18/daemon/images/image_pull.go#L54
    https://github.com/moby/moby/blob/v20.10.18/daemon/images/image_pull.go#L130
    https://github.com/moby/moby/blob/v20.10.18/distribution/pull.go#L52

    Most of the relevant code is in distribution/pull_v2.go (the high-level part).

    The code that does the http requests is in vendor/github.com/docker/distribution/registry/client/auth/session.go and vendor/github.com/docker/distribution/registry/client/repository.go.

    docker save:

    https://github.com/moby/moby/blob/v20.10.18/api/server/router/image/image.go#L31
    https://github.com/moby/moby/blob/v20.10.18/api/server/router/image/image_routes.go#L160
    https://github.com/moby/moby/blob/v20.10.18/daemon/images/image_exporter.go#L16
    https://github.com/moby/moby/blob/v20.10.18/image/tarexport/save.go#L187

    Most of the code is in image/tarexport/save.go.

    But well, apparently docker-drag‘s developer didn’t concern himself with it, and it (docker-drag) seems to work.

    In case of private Docker Hub repositories you probably need to add -u user:password to the $realm?service=$service&scope=$scope request.

    A couple of remarks about using the API:

    • A registry is a collection of repositories. Think GitHub. Generally repository names are of the form user/name (e.g. nginxproxy/nginx-proxy). But with official repositories (that all in fact start with library/) the first segment may be omitted (library/ruby -> ruby). Also there might be just one segment (e.g. in private registries), and occasionaly the first part alone is called a repository (example). Or the second.
    • A repository is a collection of images. Some of them are tagged. Untagged images usually come into being when you push a new version of an image using the same tag.
    • An old, but maybe somewhat up-to-date relevant article (at least the beginning).
    • For the Docker Hub’s registry URL see this answer.
    • Don’t expect everything in the spec to work. E.g. Docker Hub doesn’t implement the /v2/_catalog route.
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search