Question posted in
The official documentation can be found here.

Docker – Nexus Repository: BLOB_UNKNOWN: blob unknown to registry

  • The developers regularly worked with the group docker repository. And one day a complaint was received that push images were not possible.
  • When building a container, a group repository is used, which includes: docker-hosted(local), hub.docker(proxy), gcr.io(proxy), mcr.microsoft.com(proxy).
  • Logs during push:
2024-01-16 13:27:51,999+0000 ERROR [qtp69617637-29347] user-git org.sonatype.nexus.repository.docker.internal.orient.V2ManifestUtilImpl - Manifest refers to missing layer: sha256:0ec0ce3058439b66991147caf6fd33b737d8f06dec9c07885ecf836cae3d2cdd for: project/name/1.0.37404 in repository RepositoryImpl$$EnhancerByGuice$$698854289{type=hosted, format=docker, name='project-docker-hosted'}
2024-01-16 13:27:52,012+0000 ERROR [qtp69617637-29347] user-git org.sonatype.nexus.repository.docker.internal.orient.V2ManifestUtilImpl - Manifest refers to missing layer: sha256:c78ba7738ca3a1b0a85ee1fab5a5e8dbfcef6fdb5e515d748e168ce7e804a542 for: project/name/1.0.37404 in repository RepositoryImpl$$EnhancerByGuice$$698854289{type=hosted, format=docker, name='project-docker-hosted'}
2024-01-16 13:27:52,015+0000 ERROR [qtp69617637-29347] user-git org.sonatype.nexus.repository.docker.internal.orient.V2ManifestUtilImpl - Manifest refers to missing layer: sha256:7a3506e858bfcef5d2ea86c950664bdc6cf6c5ef6f278abd2d31f3285ea1e5d2 for: project/name/1.0.37404 in repository RepositoryImpl$$EnhancerByGuice$$698854289{type=hosted, format=docker, name='project-docker-hosted'}
2024-01-16 13:27:52,028+0000 WARN  [qtp69617637-29347] user-git org.sonatype.nexus.repository.docker.internal.V2Handlers - Error: PUT /v2/project/name/manifests/1.0.37404: 400 - org.sonatype.nexus.repository.docker.internal.V2Exception: Invalid Manifest

I checked the problematic blobs. And they are indeed missing from the docker-hosted repository, however I was able to find them in mcr.microsoft.com(proxy). And I can’t understand why when pushing the problematic blobs are not sent to the hosted-repository…

Push is initially performed by gitlab-runner. I tried manual push, but it didn’t help. I also tried rebuild index and invalidate cache for docker-hosted and docker-group repositories respectively. I tried cleaning the nexus using tasks. I also tried downloading the nginx:latest image, changing the tag and uploading it to the docker-hosted repository. But I get an error:

The push refers to repository [project-docker-hosted/nginx]
009507b85609: Layer already exists 
fbcc9bc44d3e: Layer already exists 
b4ad47845036: Layer already exists 
eddcd06e5ef9: Layer already exists 
b61d4b2cd2da: Layer already exists 
b6c2a8d6f0ac: Layer already exists 
571ade696b26: Layer already exists 
errors:
blob unknown: blob unknown to registry
blob unknown: blob unknown to registry
blob unknown: blob unknown to registry
blob unknown: blob unknown to registry
blob unknown: blob unknown to registry
blob unknown: blob unknown to registry
blob unknown: blob unknown to registry
blob unknown: blob unknown to registry
Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    I was using nginx as a reverse proxy and the problem was the distribution of requests. In my original implementation of the config there was only one location ~ ^/(v1|v2)/ and redirection took place in it depending on the type:

    map $request_method $redirection {
    default                    "nexus";
    "~(GET|HEAD)"              "docker-group";
    "~(PATCH|PUT|POST|DELETE)" "docker-hosted";
}

    However, in this case, requests of type HEAD are ALWAYS redirected to the group repository, which is why the error occurred when pushing. Because blob was present in the group repository, but not in the local one.

    To solve the problem, I added another map and location.

    map:

    map $request_method $redirection {
    default                    "nexus";
    "~(GET|HEAD)"              "docker-group";
    "~(PATCH|PUT|POST|DELETE)" "docker-hosted";
}

map $request_method $redirection-blobs {
    default                         "nexus";
    "~(GET)"                        "docker-group";
    "~(PATCH|PUT|POST|DELETE|HEAD)" "docker-hosted";
}

    location:

       location ~ ^/(v1|v2)/[^/]+/?[^/]+/blobs/ {
     proxy_set_header Host $host:$server_port;
     proxy_set_header X-Real-IP $remote_addr;
     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
     proxy_set_header X-Forwarded-Proto "https";

     proxy_pass http://$redirection-blobs;
    }

   location ~ ^/(v1|v2)/ {
     proxy_set_header Host $host:$server_port;
     proxy_set_header X-Real-IP $remote_addr;
     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
     proxy_set_header X-Forwarded-Proto "https";

     proxy_pass http://$redirection-main;
    }

    After which he pushed successfully!


  2. 0

    You mentioned you’re pushing to a group repository. When docker pushes an image to a registry it checks to see if each manifest and layer in the image already exists in the remote before uploading it. It will only push each file when it has determined it doesn’t exist on the remote registry. When you push an image to a group repository it will check all group members for the incoming manifest or layer. Some of your image’s layers exist in the mcr.microsoft.com proxy repository, so they won’t be uploaded.

    If you want to ensure all of your image’s contents are in the hosted repository you need to push the image directly to that repository. Don’t use a group repository.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search