Amazon web services - Integration testing on SQS and DynamoDB with LocalStack and TestContainers runs with success locally but returns 404 in GitLab Pipelines

EdMcLaughlin
January 8, 2025
205 views
0 votes
2 Answers

As you will see below, I have integration tests running with LocalStack and TestContainers. Locally, the tests run end-to-end successfully. In GitLab Pipelines, they don’t get past the configurations, where a table and queue are created on application start.

The errors are Error creating queue: Service returned HTTP status code 404 (Service: Sqs, Status Code: 404, Request ID: null) which indicates to me that the request is malformed in some way when running in pipeline, but I can’t seem to get much insight into what the request looks like when it fails. The LocalStack logs aren’t capturing the attempted call on pipeline fails, but they do log on successful runs locally.

Grateful for any insight anyone can provide.

Gitlab CICD config:

stages: # List of stages for jobs, and their order of execution
  - build
  - test
  - deploy

image: maven:3.9.9-eclipse-temurin-21

integration-test-job:
  stage: test
  variables:
    DOCKER_HOST: tcp://docker:2375
    DOCKER_TLS_CERTDIR: ""
    DOCKER_DRIVER: overlay2
  script:
    - ./mvnw -s mvn_ci_settings.xml integration-test
  rules:
    # Run only ONCE on every commit or merge request
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    - if: $CI_COMMIT_BRANCH
  artifacts:
    reports:
      junit: target/surefire-reports/TEST-*.xml

This test configuration creates the localStackContainer Bean

@TestConfiguration(proxyBeanMethods = false)
public class LocalStackConfig {

  @Value("${amazon.region}")
  private String awsRegion;

  private static final DockerImageName LOCALSTACK_IMAGE_NAME = DockerImageName.parse("localstack/localstack:3.8.1");

  private LocalStackContainer localStackContainer;

  @Bean
  public LocalStackContainer localStackContainer() {
    // Setting defaults for LocalStackContainer. Access and Secret keys are fake.
    // https://docs.localstack.cloud/references/credentials/
    this.localStackContainer = new LocalStackContainer(LOCALSTACK_IMAGE_NAME)
        .withEnv("LS_LOG", "trace-internal")
        .withEnv("AWS_DEFAULT_REGION", awsRegion)
        .withEnv("AWS_ACCESS_KEY_ID", "fake")
        .withEnv("AWS_SECRET_ACCESS_KEY", "fake")
        .withServices(DYNAMODB,SQS);
    return this.localStackContainer;
  }

  @PreDestroy
  public void closeLocalStackContainer() {
    if (this.localStackContainer != null) {
      this.localStackContainer.close();
    }
  }

}

This test configuration creates SQS and DynamoDB clients, creates a table and a queue on application starts, and logs LocalStack health checks.

@TestConfiguration(proxyBeanMethods = false)
public class XadsIntegrationTestConfig {

    @Autowired
    private LocalStackContainer localStackContainer;

    @Value("${amazon.sqs.name}")
    private String queueName;

    @Value("${amazon.region}")
    private String awsRegion;

    private static final Logger LOG = System.getLogger(ConsentRepositoryImpl.class.getName());

    @Bean
    public SqsClient sqsClient() {
        LOG.log(System.Logger.Level.INFO, "SQS endpoint: " + localStackContainer.getEndpointOverride(SQS));

        return SqsClient.builder()
            .region(Region.of(awsRegion))
            .credentialsProvider(
                StaticCredentialsProvider.create(
                    AwsBasicCredentials.create(localStackContainer.getAccessKey(), localStackContainer.getSecretKey())
                )
            )
            .endpointOverride(localStackContainer.getEndpointOverride(SQS))
            .build();
    }

    @Bean
    public DynamoDbClient dynamoDbClient() {
        LOG.log(System.Logger.Level.INFO, "DynamoDB endpoint: " + localStackContainer.getEndpointOverride(DYNAMODB));

        return DynamoDbClient.builder()
            .region(Region.of(awsRegion))
            .credentialsProvider(
                StaticCredentialsProvider.create(
                    AwsBasicCredentials.create(localStackContainer.getAccessKey(), localStackContainer.getSecretKey())
                )
            )
            .endpointOverride(localStackContainer.getEndpointOverride(DYNAMODB))
            .build();
    }

    @EventListener
    public void onApplicationReady(ApplicationReadyEvent applicationReadyEvent) {
        checkLSHealth();

        ApplicationContext applicationContext = applicationReadyEvent.getApplicationContext();

        LOG.log(System.Logger.Level.INFO, "Creating tables and sqsclient for integration tests");

        // Create DynamoDB table for testing
        LOG.log(System.Logger.Level.INFO, "Getting consentTable bean...");
        DynamoDbTable<Consent> consentTable = applicationContext.getBean("consentTable", DynamoDbTable.class);

        try {
            consentTable.createTable();
            LOG.log(System.Logger.Level.INFO, "consentTable created...");
            String logs = localStackContainer.getLogs();
            LOG.log(System.Logger.Level.ERROR, logs);
        } catch (Exception e) {
            String logs = localStackContainer.getLogs();
            LOG.log(System.Logger.Level.ERROR, "Error creating queue: " + e.getMessage());
            LOG.log(System.Logger.Level.ERROR, logs);
        }

        // Create SQS queue for testing
        LOG.log(System.Logger.Level.INFO, "Getting sqsclient bean...");
        SqsClient sqsClient = applicationContext.getBean("sqsClient", SqsClient.class);

        CreateQueueRequest createQueueRequest = CreateQueueRequest.builder()
            .queueName(queueName)
            .build();

        try {
            CreateQueueResponse createQueueResponse = sqsClient.createQueue(createQueueRequest);
            LOG.log(System.Logger.Level.INFO, "Queue created: " + createQueueResponse.queueUrl());
            String logs = localStackContainer.getLogs();
            LOG.log(System.Logger.Level.ERROR, logs);
        } catch (Exception e) {
            String logs = localStackContainer.getLogs();
            LOG.log(System.Logger.Level.ERROR, "Error creating queue: " + e.getMessage());
            LOG.log(System.Logger.Level.ERROR, logs);
        }


    }

    private void checkLSHealth() {
        LOG.log(System.Logger.Level.INFO, "LocalStack isRunning check...");
        LOG.log(System.Logger.Level.INFO, "LocalStack is running: " + localStackContainer.isRunning());

        HttpClient client = HttpClient.newHttpClient();
        HttpRequest request = HttpRequest.newBuilder()
            .uri(URI.create(localStackContainer.getEndpointOverride(DYNAMODB) + "/_localstack/health"))
            .GET()
            .build();

        try {
            // Send the request synchronously
            HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
            LOG.log(System.Logger.Level.INFO, "LocalStack health check response: " + response.statusCode());
            LOG.log(System.Logger.Level.INFO, "LocalStack health check response body: " + response.body());
        } catch (IOException | InterruptedException e) {
            LOG.log(System.Logger.Level.ERROR, "Error during LocalStack health check: " + e.getMessage());
            Thread.currentThread().interrupt(); // Restore interrupted status
        }

        String endpoint = localStackContainer.getEndpoint().toString();
        LOG.log(System.Logger.Level.INFO, "LocalStack endpoint: " + endpoint);

        String endpointOverride = localStackContainer.getEndpointOverride(DYNAMODB).toString();
        LOG.log(System.Logger.Level.INFO, "LocalStack endpointOverride: " + endpointOverride);


    }

}

Running locally, my tests (not shown here) run successfully. Running in GitLab Pipelines, I get 404 errors on calls to DynamoDB and SQS.

Here are logs of the run on local and in pipeline. I’m excluding the output of the LocalStack Logging below for brevity.

Local Logs:

2025-01-07T00:57:19.728-05:00  INFO 9560 --- [           main] g.f.x.p.r.impl.ConsentRepositoryImpl     : LocalStack isRunning check...
2025-01-07T00:57:19.740-05:00  INFO 9560 --- [           main] g.f.x.p.r.impl.ConsentRepositoryImpl     : LocalStack is running: true
2025-01-07T00:57:21.952-05:00  INFO 9560 --- [           main] g.f.x.p.r.impl.ConsentRepositoryImpl     : LocalStack health check response: 200
2025-01-07T00:57:21.952-05:00  INFO 9560 --- [           main] g.f.x.p.r.impl.ConsentRepositoryImpl     : LocalStack health check response body: {"services": {"acm": "disabled", "apigateway": "disabled", "cloudformation": "disabled", "cloudwatch": "disabled", "config": "disabled", "dynamodb": "available", "dynamodbstreams": "available", "ec2": "disabled", "es": "disabled", "events": "disabled", "firehose": "disabled", "iam": "disabled", "kinesis": "available", "kms": "disabled", "lambda": "disabled", "logs": "disabled", "opensearch": "disabled", "redshift": "disabled", "resource-groups": "disabled", "resourcegroupstaggingapi": "disabled", "route53": "disabled", "route53resolver": "disabled", "s3": "disabled", "s3control": "disabled", "scheduler": "disabled", "secretsmanager": "disabled", "ses": "disabled", "sns": "disabled", "sqs": "available", "ssm": "disabled", "stepfunctions": "disabled", "sts": "disabled", "support": "disabled", "swf": "disabled", "transcribe": "disabled"}, "edition": "community", "version": "3.8.1"}
2025-01-07T00:57:21.952-05:00  INFO 9560 --- [           main] g.f.x.p.r.impl.ConsentRepositoryImpl     : LocalStack endpoint: http://127.0.0.1:33121
2025-01-07T00:57:21.952-05:00  INFO 9560 --- [           main] g.f.x.p.r.impl.ConsentRepositoryImpl     : LocalStack endpointOverride: http://127.0.0.1:33121
2025-01-07T00:57:21.952-05:00  INFO 9560 --- [           main] g.f.x.p.r.impl.ConsentRepositoryImpl     : Creating tables and sqsclient for integration tests
2025-01-07T00:57:21.952-05:00  INFO 9560 --- [           main] g.f.x.p.r.impl.ConsentRepositoryImpl     : Getting consentTable bean...
2025-01-07T00:57:25.114-05:00  INFO 9560 --- [           main] g.f.x.p.r.impl.ConsentRepositoryImpl     : consentTable created...
.
.
.
2025-01-07T00:57:25.144-05:00  INFO 9560 --- [           main] g.f.x.p.r.impl.ConsentRepositoryImpl     : Getting sqsclient bean...
2025-01-07T00:57:25.243-05:00  INFO 9560 --- [           main] g.f.x.p.r.impl.ConsentRepositoryImpl     : Queue created: http://127.0.0.1:33121/queue/us-east-1/000000000000/xads-data-sharing-queue

GitLab Pipeline logs:

2025-01-07T05:40:53.906Z  INFO 179 --- [           main] g.f.x.p.r.impl.ConsentRepositoryImpl     : DynamoDB endpoint: http://172.17.0.1:32781
2025-01-07T05:40:54.782Z  INFO 179 --- [           main] g.f.x.p.r.impl.ConsentRepositoryImpl     : SQS endpoint: http://172.17.0.1:32781
2025-01-07T05:40:55.488Z  INFO 179 --- [           main] g.f.x.q.DataSharingQueueIntegrationTests : Started DataSharingQueueIntegrationTests in 22.396 seconds (process running for 23.721)
2025-01-07T05:40:55.493Z  INFO 179 --- [           main] g.f.x.p.r.impl.ConsentRepositoryImpl     : LocalStack isRunning check...
2025-01-07T05:40:55.510Z  INFO 179 --- [           main] g.f.x.p.r.impl.ConsentRepositoryImpl     : LocalStack is running: true
2025-01-07T05:40:58.925Z  INFO 179 --- [           main] g.f.x.p.r.impl.ConsentRepositoryImpl     : LocalStack health check response: 200
2025-01-07T05:40:58.926Z  INFO 179 --- [           main] g.f.x.p.r.impl.ConsentRepositoryImpl     : LocalStack health check response body: {"services": {"acm": "disabled", "apigateway": "disabled", "cloudformation": "disabled", "cloudwatch": "disabled", "config": "disabled", "dynamodb": "available", "dynamodbstreams": "available", "ec2": "disabled", "es": "disabled", "events": "disabled", "firehose": "disabled", "iam": "disabled", "kinesis": "available", "kms": "disabled", "lambda": "disabled", "logs": "disabled", "opensearch": "disabled", "redshift": "disabled", "resource-groups": "disabled", "resourcegroupstaggingapi": "disabled", "route53": "disabled", "route53resolver": "disabled", "s3": "disabled", "s3control": "disabled", "scheduler": "disabled", "secretsmanager": "disabled", "ses": "disabled", "sns": "disabled", "sqs": "available", "ssm": "disabled", "stepfunctions": "disabled", "sts": "disabled", "support": "disabled", "swf": "disabled", "transcribe": "disabled"}, "edition": "community", "version": "3.8.1"}
2025-01-07T05:40:58.927Z  INFO 179 --- [           main] g.f.x.p.r.impl.ConsentRepositoryImpl     : LocalStack endpoint: http://172.17.0.1:32781
2025-01-07T05:40:58.927Z  INFO 179 --- [           main] g.f.x.p.r.impl.ConsentRepositoryImpl     : LocalStack endpointOverride: http://172.17.0.1:32781
2025-01-07T05:40:58.927Z  INFO 179 --- [           main] g.f.x.p.r.impl.ConsentRepositoryImpl     : Creating tables and sqsclient for integration tests
2025-01-07T05:40:58.927Z  INFO 179 --- [           main] g.f.x.p.r.impl.ConsentRepositoryImpl     : Getting consentTable bean...
2025-01-07T05:40:59.643Z ERROR 179 --- [           main] g.f.x.p.r.impl.ConsentRepositoryImpl     : Error creating queue: Service returned HTTP status code 404 (Service: DynamoDb, Status Code: 404, Request ID: null)
.
.
.
025-01-07T05:40:59.646Z  INFO 179 --- [           main] g.f.x.p.r.impl.ConsentRepositoryImpl     : Getting sqsclient bean...
2025-01-07T05:41:13.994Z ERROR 179 --- [           main] g.f.x.p.r.impl.ConsentRepositoryImpl     : Error creating queue: Service returned HTTP status code 404 (Service: Sqs, Status Code: 404, Request ID: null)

Answers

- fa44
- January 7, 2025 at 9:01 pm
- 0 votes
0
Can you try to initialize your clients with localhost.localstack.cloud enpoint like suggesting in this answer?

Login or Signup to reply.

- fa44
- January 8, 2025 at 7:41 pm
- 0 votes
0
You should add following lines in your Gitlab CI/CD configuration file to run the job as a Docker container:
```
integration-test-job:
    # DinD service is required for Testcontainers
    services:
      - name: docker:dind
        # explicitly disable tls to avoid docker startup interruption
        command: ["--tls=false"]
```
From testcontainers Gitlab CI documentation:

In order to use Testcontainers in a Gitlab CI pipeline, you need to
run the job as a Docker container (see Patterns for running inside
Docker). So edit your .gitlab-ci.yml to include the Docker-In-Docker
service (docker:dind) and set the DOCKER_HOST variable to
tcp://docker:2375 and DOCKER_TLS_CERTDIR to empty string.
Login or Signup to reply.

Please signup or login to give your own answer.

Click here to cancel reply.

Amazon web services – Integration testing on SQS and DynamoDB with LocalStack and TestContainers runs with success locally but returns 404 in GitLab Pipelines

Answers