Docker – VS Code Dev Container – very slow start

I’m using VS Code with docker and Dev Container extension.
It is working fine but every time I start it I see (among other log lines)

[1597 ms] Start: Run: docker -v
[1645 ms] Start: Resolving Remote
[136401 ms] Failed to fetch control manifest: 
[136404 ms] Start: Run: docker compose version --short

I have the following config for docker proxy

$ cat ~/.docker/config.json
{
 "proxies": {
   "default": {
     "httpProxy": "http://proxy.org.com:81",
     "httpsProxy": "http://proxy.org.com:81",
     "noProxy": ".org.com"
   }
 }
}

How can I fix this so it will be able to fetch whatever it needs and start faster?

I don’t see the same thing when running the same docker container from bash in WSL.