Question 324404 in Docker Question posted in
The official documentation can be found here.

How to make https requests to a web app running on docker containers in a gitlab CI/CD pipeline?

I am trying to run some security tests agains a web app I have running in a gitlab pipeline. The app critically needs Semgrep, Docker Compose and some custom security software to run.

Here is my pipeline file https://gitlab.com/alex-nightvision1/java-github-actions-demo/-/blob/main/.gitlab-ci.yml?ref_type=heads

All relevant software can be downloaded and run with this pipeline. However, when I try to add tests within the pipeline steps that can make basic https requests like ping or curl it seems there’s something wrong with the networking component.

How do I add something so my steps can connect and make https requests to an app running in a docker container on https://127.0.0.1:9000 within the pipeline? Is it something I need to do with the /etc/hosts file?

Example output of my current job:

https://gitlab.com/alex-nightvision1/java-github-actions-demo/-/jobs/5798178141#L1655-L1764

$ docker-compose up -d
Creating network "java-github-actions-demo_default" with the default driver
Pulling db (postgres:)...
latest: Pulling from library/postgres
Digest: sha256:01cc58ff77e6e4e21999c8afff1b21954a131054c48db5269a8e6589701cc3ba
Status: Downloaded newer image for postgres:latest
Building javavulny
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            Install the buildx component to build images with BuildKit:
            https://docs.docker.com/go/buildx/
Step 1/8 : FROM openjdk:11.0.10-jdk-slim
11.0.10-jdk-slim: Pulling from library/openjdk
f7ec5a41d630: Pulling fs layer
faf4c47c8c61: Pulling fs layer
810072571faf: Pulling fs layer
4b3fe8e4a3b9: Pulling fs layer
4b3fe8e4a3b9: Waiting
810072571faf: Verifying Checksum
810072571faf: Download complete
faf4c47c8c61: Verifying Checksum
faf4c47c8c61: Download complete
f7ec5a41d630: Verifying Checksum
f7ec5a41d630: Download complete
4b3fe8e4a3b9: Verifying Checksum
4b3fe8e4a3b9: Download complete
f7ec5a41d630: Pull complete
faf4c47c8c61: Pull complete
810072571faf: Pull complete
4b3fe8e4a3b9: Pull complete
Digest: sha256:0c1702c4b90c6148a9856d3154963b7499eb9efca1687ad544340f1542c85b9f
Status: Downloaded newer image for openjdk:11.0.10-jdk-slim
 ---> b4517d9514cb
Step 2/8 : RUN mkdir /javavulny /app
 ---> Running in 6508e0856116
Removing intermediate container 6508e0856116
 ---> e77e2f1d8ce4
Step 3/8 : COPY . /javavulny/
 ---> 0b249427c4c3
Step 4/8 : RUN sed -i 's/localhost:5432/db:5432/' /javavulny/src/main/resources/application-postgresql.properties
 ---> Running in af2118e831e2
Removing intermediate container af2118e831e2
 ---> 615f30a3ce47
Step 5/8 : RUN cd /javavulny && ./gradlew --no-daemon build && cp build/libs/java-spring-vuly-0.1.0.jar /app/ && rm -Rf build/ && cd / && rm -Rf /javavulny /root/.gradle/
 ---> Running in f15e4f80e690
Downloading https://services.gradle.org/distributions/gradle-7.5.1-bin.zip
...........10%............20%...........30%............40%...........50%............60%...........70%............80%...........90%............100%
Welcome to Gradle 7.5.1!
Here are the highlights of this release:
 - Support for Java 18
 - Support for building with Groovy 4
 - Much more responsive continuous builds
 - Improved diagnostics for dependency resolution
For more details see https://docs.gradle.org/7.5.1/release-notes.html
To honour the JVM settings for this build a single-use Daemon process will be forked. See https://docs.gradle.org/7.5.1/userguide/gradle_daemon.html#sec:disabling_the_daemon.
Daemon will be stopped at the end of the build 
> Task :compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: /javavulny/src/main/java/hawk/controller/PayloadController.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
> Task :processResources
> Task :classes
> Task :bootJar
> Task :distTar NO-SOURCE
> Task :distZip NO-SOURCE
> Task :jar SKIPPED
> Task :assemble
> Task :compileTestJava NO-SOURCE
> Task :processTestResources NO-SOURCE
> Task :testClasses UP-TO-DATE
> Task :test NO-SOURCE
> Task :check UP-TO-DATE
> Task :build
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD SUCCESSFUL in 43s
3 actionable tasks: 3 executed
Removing intermediate container f15e4f80e690
 ---> dc9224c8aa07
Step 6/8 : WORKDIR /app
 ---> Running in 8f73c46d7e85
Removing intermediate container 8f73c46d7e85
 ---> 7c47b2079eb0
Step 7/8 : ENV PWD=/app
 ---> Running in c4445c76b2f5
Removing intermediate container c4445c76b2f5
 ---> 013cce8ec634
Step 8/8 : CMD ["java", "-Djava.security.egd=file:/dev/./urandom", "-jar", "/app/java-spring-vuly-0.1.0.jar"]
 ---> Running in 7cf06d01d1e9
Removing intermediate container 7cf06d01d1e9
 ---> 7bcef5d3b110
Successfully built 7bcef5d3b110
Successfully tagged stackhawk/javavulny:latest
Image for service javavulny was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Creating java-github-actions-demo_db_1 ... 
Creating java-github-actions-demo_db_1 ... done
Creating javavulny                     ... 
Creating javavulny                     ... done
$ sleep 10
$ docker ps
CONTAINER ID   IMAGE                        COMMAND                  CREATED          STATUS          PORTS                                       NAMES
fa34b64d241a   stackhawk/javavulny:latest   "java -Djava.securit…"   10 seconds ago   Up 10 seconds   0.0.0.0:9000->9000/tcp, :::9000->9000/tcp   javavulny
cec9a90873de   postgres                     "docker-entrypoint.s…"   10 seconds ago   Up 10 seconds   0.0.0.0:5432->5432/tcp, :::5432->5432/tcp   java-github-actions-demo_db_1
$ docker network ls
NETWORK ID     NAME                               DRIVER    SCOPE
776da0ca6ccd   bridge                             bridge    local
45e1bf558a3e   host                               host      local
ad8c9d54c9f7   java-github-actions-demo_default   bridge    local
bbf033f8208c   none                               null      local
$ curl -k https://127.0.0.1:9000
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9000 after 0 ms: Connection refused

Here is the output.

Tags:

2

Answers


  1. 0

    You are using docker:dind to build your image and run at runner, and your runner execution base is ubuntu:latest

    I’m little confused, but I think if you use
    "curl localhost or 127.0.0.1 or https://docker",
    that command use ubuntu:latest

    But maybe your container with image stackhawk/javavulny:latest is located on Dind ( isolated from host )

    So maybe you can try change test image to ubuntu:latest -> docker:latest and retry ( to mount docker socket to host )

    References

    https://forum.gitlab.com/t/why-services-docker-dind-is-needed-while-already-having-image-docker/43534

    Login or Signup to reply.
  2. 0

    I think your configuration has 2 issues:

    • curl is using localhost instead of docker to connect to your container (though it seems you fixed it recently). You should use curl -k https://docker:9000
      • You are running a docker:dind service, and your app container runs inside this other container. docker service is reachable via docker hostname from your CI job, and your app’s container is running inside this container, listening on port 9000
      • See this question and GitLab doc on accessing service
    • curl ...:9000 is probably running before your app is completely started (it does not yet listen on port 9000), hence the connection refused. Spring is known to be slow to start, this is confirmed by logs in your pipelines where typical "startup OK" lines like Tomcat started or Started Application are not shown. Spring apps will typically show logs like this after successful startup: 
      2024-01-20 10:00:04.234  INFO 9999 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 9000 (http) with context path ''
2024-01-20 10:00:04.345  INFO 9999 --- [           main] com.example.demo.Application             : Started Application in 40.420 seconds (JVM running for 40.475)

    You could sleep longer before running curl but it’s gonna be painful to wait on each pipelines. A better fail-fast approach is to leverage docker compose --wait and healthcheck on your app container to ensure your application is healthy (fully started) before running your tests:

    • Update your docker-compose.yml with a healthcheck such as 
        javavulny:
    # ...
    # Wait up to 3 min for app to start 
    # and be responsive on port 9000
    healthcheck:
      test: ["CMD", "curl", "-k", "https://localhost:9000"]
      interval: 10s
      timeout: 5s
      retries: 18
    • Wait for successful startup with docker-compose up -d --wait and remove sleep 10
    • Use curl -k https://docker:9000 instead of curl -k https://localhost:9000 to reach the DinD container where your app container is running

    Now, Docker Compose will wait for app to be ready and accepting requests on port 9000. Your subsequent curl -k https://docker:9000 will likely succeed.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search