Debian Question posted in
The oficial documentation can be found here.

Using docker Container links in Jenkins – Debian

I am trying to run a CI system which works for DB migration scripts. So this could prevent us from running sql scripts which do not work at the migration time because of syntax errors. For our local setup we use docker-compose and run multiple containers at a time. This is unfortunately not an option for us in Jenkins pipeline

I created this Jenkinsfile with the following strategy.

  1. A postgres container is run with more or less default parameters
  2. Another postgres container is linked to the first container, only to use pg_isready commandline to wait until the DB is ready to take connections
  3. Use flyway container to run DB migrations against the DB setup in step 1. Eventually planned to run E2E tests also with web applications

My implementation is based on the documentation here (running docker sidecar). However, this is not working and the first container (step 1) seems to be stopping. I added some additional debugging (try catch) to see the logs of this container

Content of my Jenkinsfile

def docker_repository = '<CUSTOM-REGISTRY>'
def docker_user_credentialsId = '<DOCKER-USER>'

pipeline {
   agent { label 'docker && linux && nonprod' }

options {
    buildDiscarder(logRotator(daysToKeepStr: '90', numToKeepStr: '20', artifactDaysToKeepStr: '90', artifactNumToKeepStr: '20'))
    timeout(time: 20, unit: 'MINUTES') 
}
stages {
    stage('build & test') {
        environment {
            POSTGRES_DB = 'mydb'
            POSTGRES_USER = 'postgres'
            POSTGRES_PASSWORD = 'postgres'                
            FLYWAY_URL = 'jdbc:postgresql://localhost:5432/mydb'
            // FLYWAY_URL = 'jdbc:postgresql://db-container:5432/mydb'
            FLYWAY_USER = 'postgres'
            FLYWAY_PASSWORD = 'postgres'
        }
        steps {
            checkout scm
            withCredentials([ usernamePassword(credentialsId: docker_user_credentialsId, passwordVariable: 'ARTIFACTORY_API_TOKEN', usernameVariable: 'ARTIFACTORY_API_USER'),]) {}                                
            script {
                docker.withRegistry(docker_repository, docker_user_credentialsId) {
                    docker.image('<REGISTRY>/postgres').withRun("--name=db-container -e POSTGRES_PASSWORD=postgres") { c ->
                        try {
                            docker.image('<REGISTRY>/postgres').inside("--link ${c.id}:db") {                                
                                sh '''
                                    while ! pg_isready -h db -p 5432
                                    do
                                         echo $
                                         echo "$(date) - waiting for database to start"
                                         sleep 10
                                    done
                                '''
                            }
                            docker.image('<REGISTRY>/flyway/flyway').inside("--link ${c.id}:db") {
                                sh 'info'
                                // sh 'migrate'
                            }
                        } catch (exc) {
                            sh "docker logs ${c.id}"
                            throw exc
                        }
                    }
                }
            }                
        }
    }
}
post {
    always {
        cleanWs()
    }
}

}

Step 2 was temporarily removed to get this logs since step 2 waits in a loop and the docker logs

+ docker logs 9d9e8699b57430e288520c485c8333a0261f9283f749aec2832cfb0e5f19ef9e
 The files belonging to this database system will be owned by user "postgres".
 This user must also own the server process.
 The database cluster will be initialized with locale "en_US.utf8".
 The default database encoding has accordingly been set to "UTF8".
 The default text search configuration will be set to "english".

 Data page checksums are disabled.
 fixing permissions on existing directory /var/lib/postgresql/data ... ok
  creating subdirectories ... ok
  selecting dynamic shared memory implementation ... posix
  selecting default max_connections ... 100
  selecting default shared_buffers ... 128MB 
  selecting default time zone ... Etc/UTC
  creating configuration files ... ok
  running bootstrap script ... ok
  performing post-bootstrap initialization ... ok
  syncing data to disk ... ok
  Success. You can now start the database server using:
  pg_ctl -D /var/lib/postgresql/data -l logfile start
  initdb: warning: enabling "trust" authentication for local connections
  You can change this by editing pg_hba.conf or using the option -A, or
   --auth-local and --auth-host, the next time you run initdb.
  waiting for server to start....2021-03-31 15:21:29.923 UTC [48] LOG:  starting PostgreSQL 13.2 
  (Debian 13.2-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
  2021-03-31 15:21:29.929 UTC [48] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
  2021-03-31 15:21:29.946 UTC [49] LOG:  database system was shut down at 2021-03-31 15:21:29 UTC
  2021-03-31 15:21:29.951 UTC [48] LOG:  database system is ready to accept connections
  done

  server started

   /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
   2021-03-31 15:21:30.062 UTC [48] LOG:  received fast shutdown request
   waiting for server to shut down....2021-03-31 15:21:30.064 UTC [48] LOG:  aborting any active transactions

   2021-03-31 15:21:30.065 UTC [48] LOG:  background worker "logical replication launcher" (PID 55) 
     exited with exit code 1

    2021-03-31 15:21:30.071 UTC [50] LOG:  shutting down
    2021-03-31 15:21:30.099 UTC [48] LOG:  database system is shut down
    done

    server stopped
    PostgreSQL init process complete; ready for start up.

    2021-03-31 15:21:30.188 UTC [1] LOG:  starting PostgreSQL 13.2 (Debian 13.2-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit

    2021-03-31 15:21:30.190 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
    2021-03-31 15:21:30.190 UTC [1] LOG:  listening on IPv6 address "::", port 5432
    2021-03-31 15:21:30.196 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
    2021-03-31 15:21:30.203 UTC [67] LOG:  database system was shut down at 2021-03-31 15:21:30 UTC
    2021-03-31 15:21:30.208 UTC [1] LOG:  database system is ready to accept connections

The postgres and flyway versions should not play a significant role here but as it is seen in the logs, the Postgres Version is 13.2 as provided by the default tag.

Any pointers here?

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    After some trial and errors, here is what worked for us. Although it is very similar to the initial set up, the key change is at the choice of using inside and withRun. The final Jenkinsfile looks like this.

    // credentials
def docker_user_credentialsId = '<DOCKER-USER>'
def docker_repository = '<CUSTOM-REGISTRY>'

// docker images
def pg = docker.image('postgres:12-alpine')
def flyway = docker.image('flyway/flyway:7-alpine')

pipeline {
    agent none

    options {
        buildDiscarder(logRotator(daysToKeepStr: '90', numToKeepStr: '20', artifactDaysToKeepStr: '90', artifactNumToKeepStr: '20'))
        timeout(time: 120, unit: 'MINUTES') 
    }

    stages {
        stage('db migration test') {
            agent { 
                label 'docker && linux'
            }

            environment {
                PGHOST = 'db'
                PGPORT = '5432'
                PGDATABASE = 'postgres'
                PGUSER = 'postgres'
                PGPASSWORD = 'postgres'
            }

            steps {
                script {
                    docker.withRegistry(docker_repository, docker_user_credentialsId) {
                        def db = pg.withRun(
                            "-v ${WORKSPACE}/docker/init-user-db.sh:/docker-entrypoint-initdb.d/init-user-db.sh -e POSTGRES_USER=${env.PGUSER} -e POSTGRES_PASSWORD=${env.PGPASSWORD} -e POSTGRES_DB=${env.PGDATABASE}"
                        ) { db ->
                            pg.inside("--link ${db.id}:${env.PGHOST}") {                                
                                sh '''
                                    echo "$(date) - waiting for database to start"
                                    while ! pg_isready
                                    do
                                        sleep 10
                                    done
                                '''
                            }
                            flyway.withRun("-e FLYWAY_LOCATIONS=filesystem:/tmp/database/server,filesystem:/tmp/database/local -e FLYWAY_URL=jdbc:postgresql://${PGHOST}:${env.PGPORT}/${env.PGDATABASE} -e FLYWAY_USER=${env.PGUSER} -e FLYWAY_PASSWORD=${env.PGPASSWORD} -v ${WORKSPACE}/database/local:/tmp/database/local -v ${WORKSPACE}/database/server:/tmp/database/server --link ${db.id}:db", "migrate") { f ->
                                sh "docker logs -f ${f.id}"
                                def output = sh script: "docker inspect ${f.id} --format='{{.State.ExitCode}}'", returnStdout: true
                                sh "exit ${output}"
                            }
                        }
                    }
                }
            }
        }
    }
 }

  2. 0

    A possible root cause, assuming you are using the official postgres docker image, is that the container’s entrypoint‘s script is failing.

    From the image’s Docker page:

    Warning: scripts in /docker-entrypoint-initdb.d are only run if you start the container with a data directory that is empty; […] One common problem is that if one of your /docker-entrypoint-initdb.d scripts fails (which will cause the entrypoint script to exit) and your orchestrator restarts the container with the already initialized data directory, it will not continue on with your scripts

    Solution

    Assuming that the entrypoint process is indeed failing (probably because it sees a DB that is not owned and is designed by entrypoint to override it, since it’s linked to another postgres container), you can overcome this by setting your own entrypoint and cmd, thus preventing the image from trying to create a db that you are not going to use anyway, i.e.:

    FROM postgres

#Switch back to default entrypoint
ENTRYPOINT ["/bin/sh"]

#Let the container start and do nothing until the pipeline kicks in
CMD ["-c","sleep infinity"]

    Create a new image based on above Dockerfile, push it, and define it in Jenkin’s pipeline instead of the current (only for the sidecar). You should then have a stable container, linked to the actual db container, that the pipeline can jump into.

    Finally, regarding the command launched in the sidecar, I’d suggest to use localhost instead of db as the target host, on command question’s pg_isready -h db -p 5432, reulting in: pg_isready -h localhost -p 5432

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search