Nginx Question posted in
The official documentation can be found here.

kubernetes nginx upstream server as service not found

These are my yaml files referenced by here (kubernetes.io/docs)

deployment gists

And I use command kubectl apply -f backend-deployment.yaml -f frontend-configmap.yaml -f frontend-deployment.yaml

Backend were launched successfully, but Frontend occured error.

[emerg] 1#1: host not found in upstream "backend-service" in /etc/nginx/conf.d/nginx.conf:2
#nginx: [emerg] host not found in upstream "backend-service" in /etc/nginx/conf.d/nginx.conf:2

Even if ‘backend-service’ is declared, nginx couldn’t recognize it.

Command nslookup backend-service result is:

Server:         127.0.0.53
Address:        127.0.0.53#53

** server can't find backend-service: SERVFAIL

What am I missing?

Tags:

3

Answers


  1. Chosen as BEST ANSWER
    0

    backend-deployment image : bateaux/test-nginx

    frontend-deployment image : bateaux/test-node-server

    When deploy backend-deployment, remove [livenessProbe, readinessProbe].

    Because node.js express server use @cloudnative/health-connect To readinessProbe, so call PingCheck(worker node ip).


  2. 0

    I have gotten closer to the issue/solution.

    First of all, you if the images in your supplied answer are correct, you are running nginx in your backend deployment and node-js server on your frontend deployment. This is a mistake.

    after changing the images frontend is running (as expected) but backend pod is crashing.

    However, the backend pod is successfully resolving the mysql-service to its internal clusterIp and I appear to have wrong authentication setup.

    `> [email protected] start /usr/src/app
 > node backend.js
 
 (node:18) Warning: Accessing non-existent property 'count' of module exports inside circular dependency
 (Use `node --trace-warnings ...` to show where the warning was created)
 (node:18) Warning: Accessing non-existent property 'findOne' of module exports inside circular dependency
 (node:18) Warning: Accessing non-existent property 'remove' of module exports inside circular dependency
 (node:18) Warning: Accessing non-existent property 'updateOne' of module exports inside circular dependency
 listening on 3000
 events.js:292
       throw er; // Unhandled 'error' event
       ^
 
 Error: connect ECONNREFUSED 10.100.77.32:3306
     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1142:16)
     --------------------
     at Protocol._enqueue (/usr/src/app/node_modules/mysql/lib/protocol/Protocol.js:144:48)
     at Protocol.handshake (/usr/src/app/node_modules/mysql/lib/protocol/Protocol.js:51:23)
     at Connection.connect (/usr/src/app/node_modules/mysql/lib/Connection.js:116:18)
     at Object.<anonymous> (/usr/src/app/backend.js:58:12)
     at Module._compile (internal/modules/cjs/loader.js:1185:30)
     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1205:10)
     at Module.load (internal/modules/cjs/loader.js:1034:32)
     at Function.Module._load (internal/modules/cjs/loader.js:923:14)
     at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
     at internal/main/run_main_module.js:17:47
 Emitted 'error' event on Connection instance at:
     at Connection._handleProtocolError (/usr/src/app/node_modules/mysql/lib/Connection.js:423:8)
     at Protocol.emit (events.js:315:20)
     at Protocol._delegateError (/usr/src/app/node_modules/mysql/lib/protocol/Protocol.js:398:10)
     at Handshake.<anonymous> (/usr/src/app/node_modules/mysql/lib/protocol/Protocol.js:153:12)
     at Handshake.emit (events.js:315:20)
     at Handshake.Sequence.end (/usr/src/app/node_modules/mysql/lib/protocol/sequences/Sequence.js:78:12)
     at Protocol.handleNetworkError (/usr/src/app/node_modules/mysql/lib/protocol/Protocol.js:369:14)
     at Connection._handleNetworkError (/usr/src/app/node_modules/mysql/lib/Connection.js:418:18)
     at Socket.emit (events.js:315:20)
     at emitErrorNT (internal/streams/destroy.js:96:8)
     at emitErrorCloseNT (internal/streams/destroy.js:68:3)
     at processTicksAndRejections (internal/process/task_queues.js:84:21) {
   errno: -111,
   code: 'ECONNREFUSED',
   syscall: 'connect',
   address: '10.100.77.32',
   port: 3306,
 > node backend.js
   fatal: true
 }
 npm ERR! code ELIFECYCLE
 npm ERR! errno 1
 npm ERR! [email protected] start: `node backend.js`
 npm ERR! Exit status 1
 npm ERR! 
 npm ERR! Failed at the [email protected] start script.
 npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
 
 npm ERR! A complete log of this run can be found in:
 npm ERR!     /root/.npm/_logs/2021-07-21T08_46_32_833Z-debug.log`

    if you look at the log. it says it tries to connect to 10.100.77.32:3306 and the output of kubectl get svc is the following mysql-service ClusterIP 10.100.77.32 <none> 3306/TCP 17m

    with the matching ip of my service.

    kubectl get endpoints shows that the mysql-service has indeed found an endpoint:

    mysql-service      ClusterIP   10.100.77.32     <none>        3306/TCP       17m

    and here is additional log information showing that the authentication mechanism of your nodejs application appearently is not working for the mysql pod.

      Error: ER_NOT_SUPPORTED_AUTH_MODE: Client does not support authentication protocol requested by server; consider upgrading MySQL client
    at Handshake.Sequence._packetToError (/usr/src/app/node_modules/mysql/lib/protocol/sequences/Sequence.js:47:14)
    at Handshake.ErrorPacket (/usr/src/app/node_modules/mysql/lib/protocol/sequences/Handshake.js:123:18)
    at Protocol._parsePacket (/usr/src/app/node_modules/mysql/lib/protocol/Protocol.js:291:23)
    at Parser._parsePacket (/usr/src/app/node_modules/mysql/lib/protocol/Parser.js:433:10)
    at Parser.write (/usr/src/app/node_modules/mysql/lib/protocol/Parser.js:43:10)
    at Protocol.write (/usr/src/app/node_modules/mysql/lib/protocol/Protocol.js:38:16)
    at Socket.<anonymous> (/usr/src/app/node_modules/mysql/lib/Connection.js:88:28)
    at Socket.<anonymous> (/usr/src/app/node_modules/mysql/lib/Connection.js:526:10)
    at Socket.emit (events.js:315:20)
    at addChunk (_stream_readable.js:296:12)
    at readableAddChunk (_stream_readable.js:272:9)
    at Socket.Readable.push (_stream_readable.js:213:10)
    at TCP.onStreamRead (internal/stream_base_commons.js:186:23)
    --------------------
    at Protocol._enqueue (/usr/src/app/node_modules/mysql/lib/protocol/Protocol.js:144:48)
    at Protocol.handshake (/usr/src/app/node_modules/mysql/lib/protocol/Protocol.js:51:23)
    at Connection.connect (/usr/src/app/node_modules/mysql/lib/Connection.js:116:18)
    at Object.<anonymous> (/usr/src/app/backend.js:58:12)
    at Module._compile (internal/modules/cjs/loader.js:1185:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1205:10)
    at Module.load (internal/modules/cjs/loader.js:1034:32)
    at Function.Module._load (internal/modules/cjs/loader.js:923:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
    at internal/main/run_main_module.js:17:47
Emitted 'error' event on Connection instance at:
    at Connection._handleProtocolError (/usr/src/app/node_modules/mysql/lib/Connection.js:423:8)
    at Protocol.emit (events.js:315:20)
    at Protocol._delegateError (/usr/src/app/node_modules/mysql/lib/protocol/Protocol.js:398:10)
    at Handshake.<anonymous> (/usr/src/app/node_modules/mysql/lib/protocol/Protocol.js:153:12)
    at Handshake.emit (events.js:315:20)
    at Handshake.Sequence.end (/usr/src/app/node_modules/mysql/lib/protocol/sequences/Sequence.js:78:12)
    at Handshake.ErrorPacket (/usr/src/app/node_modules/mysql/lib/protocol/sequences/Handshake.js:125:8)
    at Protocol._parsePacket (/usr/src/app/node_modules/mysql/lib/protocol/Protocol.js:291:23)
    [... lines matching original stack trace ...]
    at readableAddChunk (_stream_readable.js:272:9) {
  code: 'ER_NOT_SUPPORTED_AUTH_MODE',
  errno: 1251,
  sqlMessage: 'Client does not support authentication protocol requested by server; consider upgrading MySQL client',
  sqlState: '08004',
  fatal: true
}
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] start: `node backend.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2021-07-21T08_55_22_133Z-debug.log

    However, with the changes suggest at the beginning of the answer the issues around resolving the service name and communicating inside of kubernetes should be resolved.

    Login or Signup to reply.
  3. 0

    Wow….I found solution for getaddrinfo EAI_AGAIN mysql-service.

    I think if network can’t find service, there is error in dns.

    kubectl logs --namespace=kube-system -l k8s-app=kube-dns

    Result of command is gist.

    So…I finally found this link about kubernetes coredns.

    By belphegor’s answer, it works restart coredns.

    Although, there is new error Client does not support authentication protocol requested by server; consider upgrading MySQL client, but i think to solve it is easier than dns error. hhhhhhhhhhhhhhhh

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search