Question 212157 in Amazon Web Sevices Question posted in
The official Amazon Web Services documentation can be found here.

Amazon web services – Spring boot pod not coming up in kubernetes

I have deployed my app in aws kubernetes cluster few days back and it has been a successful deploy and the app was up and running. Recently we have changed the db password. Now there are 4 apps which connect to the same db. So we restarted all 4 apps. 3 apps came up but one of the app is not coming up. It always restarts and I have captured the below log :

[l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
[l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Fill pool skipped, pool is at sufficient level.
[nnection-reaper] h.i.c.PoolingHttpClientConnectionManager : Closing connections idle longer than 60000 MILLISECONDS
[nnection-reaper] h.i.c.PoolingHttpClientConnectionManager : Closing connections idle longer than 60000 MILLISECONDS
[alina-utility-2] org.apache.catalina.session.ManagerBase  : Start expire sessions StandardManager at 1689843028211 sessioncount 0
[alina-utility-2] org.apache.catalina.session.ManagerBase  : End expire sessions StandardManager processingTime 1 expired sessions: 0
[l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
[l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Fill pool skipped, pool is at sufficient level.
[l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
[l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Fill pool skipped, pool is at sufficient level.
[nnection-reaper] h.i.c.PoolingHttpClientConnectionManager : Closing connections idle longer than 60000 MILLISECONDS
[nnection-reaper] h.i.c.DefaultManagedHttpClientConnection : http-outgoing-1: Close connection
[nnection-reaper] com.amazonaws.internal.SdkSSLSocket      : closing ssm.us-east-1.amazonaws.com/52.46.145.233:443
[nnection-reaper] h.i.c.PoolingHttpClientConnectionManager : Closing connections idle longer than 60000 MILLISECONDS
[nnection-reaper] h.i.c.DefaultManagedHttpClientConnection : http-outgoing-0: Close connection
[nnection-reaper] com.amazonaws.internal.SdkSSLSocket      : closing sts.us-east-1.amazonaws.com/67.220.245.46:443
[alina-utility-2] org.apache.catalina.session.ManagerBase  : Start expire sessions StandardManager at 1689843088214 sessioncount 0
[alina-utility-2] org.apache.catalina.session.ManagerBase  : End expire sessions StandardManager processingTime 0 expired sessions: 0

Adding relevant hikari pool initialization log :

[           main] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Added connection org.postgresql.jdbc.PgConnection@4dd90166
[           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Start completed.
[           main] o.h.e.j.e.i.JdbcEnvironmentInitiator     : Database ->
2023-07-20T12:15:50.405833777Z stdout F        name : PostgreSQL
2023-07-20T12:15:50.405837321Z stdout F     version : 13.8
2023-07-20T12:15:50.405840197Z stdout F       major : 13
2023-07-20T12:15:50.405843007Z stdout F       minor : 8
[           main] o.h.e.j.e.i.JdbcEnvironmentInitiator     : Driver ->
2023-07-20T12:15:50.405900698Z stdout F        name : PostgreSQL JDBC Driver
2023-07-20T12:15:50.405904562Z stdout F     version : 42.3.7
2023-07-20T12:15:50.405907398Z stdout F       major : 42
2023-07-20T12:15:50.405910368Z stdout F       minor : 3
[           main] o.h.e.j.e.i.JdbcEnvironmentInitiator     : JDBC version : 4.2
[l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Pool stats (total=1, active=1, idle=0, waiting=0)
[onnection adder] org.postgresql.Driver                    : Connecting with URL: jdbc:postgresql://XXXXXX:5432/ics?useSSL=true
[onnection adder] org.postgresql.jdbc.PgConnection         : PostgreSQL JDBC Driver 42.3.7
[onnection adder] org.postgresql.jdbc.PgConnection         :   setDefaultFetchSize = 0

[onnection adder] o.p.core.v3.ConnectionFactoryImpl        : Trying to establish a protocol version 3 connection to XXXXXX
[onnection adder] o.p.core.v3.ConnectionFactoryImpl        : Receive Buffer Size is 65,536
[onnection adder] o.p.core.v3.ConnectionFactoryImpl        : Send Buffer Size is 166,400
[onnection adder] org.postgresql.ssl.MakeSSL               : converting regular socket connection to ssl
[           main] org.hibernate.dialect.Dialect            : HHH000400: Using dialect: org.hibernate.dialect.PostgreSQL10Dialect
[           main] o.h.e.j.env.spi.IdentifierHelperBuilder  : JDBC driver metadata reported database stores quoted identifiers in 
                                                              neither upper, lower nor mixed case
[onnection adder] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Added connection org.postgresql.jdbc.PgConnection@a1713f8
[onnection adder] org.postgresql.Driver                    : Connecting with URL: jdbc:postgresql://XXXXXX:5432/ics?useSSL=true
[onnection adder] org.postgresql.jdbc.PgConnection         : PostgreSQL JDBC Driver 42.3.7
[onnection adder] org.postgresql.jdbc.PgConnection         :   setDefaultFetchSize = 0
[onnection adder] org.postgresql.jdbc.PgConnection         :   setPrepareThreshold = 5
[onnection adder] o.p.core.v3.ConnectionFactoryImpl        : Trying to establish a protocol version 3 connection to XXXXXX:5432
[onnection adder] o.p.core.v3.ConnectionFactoryImpl        : Receive Buffer Size is 65,536
[onnection adder] o.p.core.v3.ConnectionFactoryImpl        : Send Buffer Size is 166,400

[           main] o.h.type.spi.TypeConfiguration$Scope     : Scoping TypeConfiguration [org.hibernate.type.spi.TypeConfiguration@7c369270] to MetadataBuildingContext [org.hibernate.boot.internal.MetadataBuildingContextRootImpl@9f86dc3]
[           main] org.hibernate.type.BasicTypeRegistry     : HHH000270: Type registration [java.util.UUID] overrides previous : org.hibernate.type.UUIDBinaryType@25f15f50
[           main] org.hibernate.type.BasicTypeRegistry     : Adding type registration java.util.UUID -> org.hibernate.type.PostgresUUIDType@4ed15347
[           main] org.hibernate.type.BasicTypeRegistry     : Adding type registration pg-uuid -> org.hibernate.type.PostgresUUIDType@4ed15347



[onnection adder] org.postgresql.ssl.MakeSSL               : converting regular socket connection to ssl

[           main] o.h.boot.model.relational.Namespace      : Created database namespace [logicalName=Name{catalog=null, schema=null}, physicalName=Name{catalog=null, schema=null}]
[onnection adder] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Added connection org.postgresql.jdbc.PgConnection@4d0fcca
[onnection adder] org.postgresql.Driver                    : Connecting with URL: jdbc:postgresql://XXXXXX:5432/ics?useSSL=true
[onnection adder] org.postgresql.jdbc.PgConnection         : PostgreSQL JDBC Driver 42.3.7
[onnection adder] org.postgresql.jdbc.PgConnection         :   setDefaultFetchSize = 0
[onnection adder] org.postgresql.jdbc.PgConnection         :   setPrepareThreshold = 5
[onnection adder] o.p.core.v3.ConnectionFactoryImpl        : Trying to establish a protocol version 3 connection to XXXXXX:5432
[onnection adder] o.p.core.v3.ConnectionFactoryImpl        : Receive Buffer Size is 65,536
[onnection adder] o.p.core.v3.ConnectionFactoryImpl        : Send Buffer Size is 166,400

[           main] org.hibernate.cfg.AnnotationBinder       : Binding entity from annotated class: x.y.z.entity.MyEntity
[onnection adder] org.postgresql.ssl.MakeSSL               : converting regular socket connection to ssl
[           main] org.hibernate.cfg.Ejb3Column             : Binding column: Ejb3DiscriminatorColumn{logicalColumnName'DTYPE', discriminatorTypeName='string'}
[           main] o.h.cfg.annotations.EntityBinder         : Import with entity name MyEntity
[           main] o.h.cfg.annotations.EntityBinder         : Bind entity x.y.z.entity.MyEntity on table my_table

So this is for sure that database connectivity is not an issue.

There is no exception and no error. When I checked the pod events I can see :

Event : Warning
Reason : Unhealthy
Message : Liveness probe failed. Get " http:// ...." dial tcp : ....:8080 connect : connection refused.
        : Readiness probe failed. Get " http:// ...." dial tcp : ....:8080 connect : connection refused.
        : Backoff restarting failed container.

This is my liveness-readiness configuration :

  - op: add
    path: /spec/template/spec/containers/0/readinessProbe/initialDelaySeconds
    value: 180
  - op: add
    path: /spec/template/spec/containers/0/livenessProbe/initialDelaySeconds
    value: 180

application properties :

management.endpoints.web.exposure.include=health,info,metrics,env,flyway
management.endpoint.health.probes.enabled=true
management.health.livenessState.enabled=true
management.health.readinessState.enabled=true

Also I have disabled the liveness and readiness but result still same.

Spring Boot version : 2.7.5

Same configuration the app is up and running in dev and preprod cluster but failing in prod cluster. Also same configuration other apps running fine. Can someone help me to identify what can be the issue here.

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    Setting the following in prod resolved the issue for me.

    spring.jpa.hibernate.ddl-auto=none

    This prevented the ddl generation most probably where the app was stuck and getting restarted.


  2. 0

    Please follow these steps to troubleshoot the issue:

    Remove the liveness and readiness probes from the deployment to narrow down the problem. Check if the pods come up after this change. If the issue is resolved, there might be a problem with the timing of your probes.

    If the previous step doesn’t resolve the issue, deploy the application in debug mode. Then, check the logs to see if it is able to acquire a JDBC connection.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search