Question 97251 in Nginx Question posted in
The official documentation can be found here.

WSO2 4.0.0 control-plane as key manager : Error while parsing JWT – Connection refused to localhost:443 – Nginx

I use two WSO2 Api Manager v4.0.0 nodes inside a private network :

  • reverse is the Nginx node that proxy requests to destination ports as a gateway/entrypoint
  • apim-bo is the WSO2 control-plane. It hosts devportal/publisher and is the key manager. IP 192.168.0.6, WSO2 listen on port 9443, domain apimbo.mysite.com
  • apim-gw is the WSO2 gateway worker used to handle requests. IP 192.168.0.5, WSO2 listen on port 8243, domain apim.mysite.com
  • I use a wildcard certificate shared between nginx and the two WSO2 nodes, same certificates and keystore files on all nodes

It works pretty much fine but I can see a lot of errors in apim-gw logs :

[2023-01-16 16:19:34,010] ERROR
{org.wso2.carbon.apimgt.impl.jwt.JWTValidatorImpl} – Error while
parsing JWT org.apache.http.conn.HttpHostConnectException: Connect to
localhost:443 [localhost/127.0.0.1] failed: Connection refused
(Connection refused) at
org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
at
org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at
org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at
org.wso2.carbon.apimgt.impl.utils.JWTUtil.retrieveJWKSConfiguration_aroundBody0(JWTUtil.java:58)
at
org.wso2.carbon.apimgt.impl.utils.JWTUtil.retrieveJWKSConfiguration(JWTUtil.java:52)
at
org.wso2.carbon.apimgt.impl.jwt.JWTValidatorImpl.retrieveJWKSet_aroundBody22(JWTValidatorImpl.java:231)
at
org.wso2.carbon.apimgt.impl.jwt.JWTValidatorImpl.retrieveJWKSet(JWTValidatorImpl.java:228)
at
org.wso2.carbon.apimgt.impl.jwt.JWTValidatorImpl.validateSignature_aroundBody8(JWTValidatorImpl.java:145)
at
org.wso2.carbon.apimgt.impl.jwt.JWTValidatorImpl.validateSignature(JWTValidatorImpl.java:135)
at
org.wso2.carbon.apimgt.impl.jwt.JWTValidatorImpl.validateToken_aroundBody0(JWTValidatorImpl.java:62)
at
org.wso2.carbon.apimgt.impl.jwt.JWTValidatorImpl.validateToken(JWTValidatorImpl.java:57)
at
org.wso2.carbon.apimgt.impl.jwt.JWTValidationServiceImpl.validateJWTToken_aroundBody0(JWTValidationServiceImpl.java:44)
at
org.wso2.carbon.apimgt.impl.jwt.JWTValidationServiceImpl.validateJWTToken(JWTValidationServiceImpl.java:36)
at
org.wso2.carbon.apimgt.gateway.handlers.security.jwt.JWTValidator.getJwtValidationInfo_aroundBody18(JWTValidator.java:519)
at
org.wso2.carbon.apimgt.gateway.handlers.security.jwt.JWTValidator.getJwtValidationInfo(JWTValidator.java:487)
at
org.wso2.carbon.apimgt.gateway.handlers.security.jwt.JWTValidator.authenticate_aroundBody0(JWTValidator.java:170)
at
org.wso2.carbon.apimgt.gateway.handlers.security.jwt.JWTValidator.authenticate(JWTValidator.java:138)
at
org.wso2.carbon.apimgt.gateway.handlers.security.oauth.OAuthAuthenticator.authenticate_aroundBody4(OAuthAuthenticator.java:323)
at
org.wso2.carbon.apimgt.gateway.handlers.security.oauth.OAuthAuthenticator.authenticate(OAuthAuthenticator.java:115)
at
org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler.isAuthenticate_aroundBody42(APIAuthenticationHandler.java:428)
at
org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler.isAuthenticate(APIAuthenticationHandler.java:422)
at
org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler.handleRequest_aroundBody36(APIAuthenticationHandler.java:353)
at
org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler.handleRequest(APIAuthenticationHandler.java:317)
at org.apache.synapse.api.API.process(API.java:389) at
org.apache.synapse.api.AbstractApiHandler.apiProcessNonDefaultStrategy(AbstractApiHandler.java:107)
at
org.apache.synapse.api.AbstractApiHandler.identifyAPI(AbstractApiHandler.java:127)
at
org.apache.synapse.api.AbstractApiHandler.dispatchToAPI(AbstractApiHandler.java:59)
at
org.apache.synapse.api.rest.RestRequestHandler.dispatchToAPI(RestRequestHandler.java:84)
at
org.apache.synapse.api.rest.RestRequestHandler.process(RestRequestHandler.java:70)
at
org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:54)
at
org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:344)
at
org.apache.synapse.core.axis2.SynapseMessageReceiver.receive(SynapseMessageReceiver.java:101)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at
org.apache.synapse.transport.passthru.ServerWorker.processNonEntityEnclosingRESTHandler(ServerWorker.java:375)
at
org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:189)
at
org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834) Caused by:
java.net.ConnectException: Connection refused (Connection refused) at
java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at
java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
at
java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
at
java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
at
java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
at java.base/java.net.Socket.connect(Socket.java:591) at
org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:368)
at
org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)

Do you have an idea of what I am missing please ?

reverse apim-bo vhost :

upstream apimbo {

    server 192.168.0.6:9443;

}

server {

    # Listen
    server_name apimbo.mysite.com;
    listen      443      ssl http2;
    listen      [::]:443 ssl http2;

    # Certs
    ssl_certificate     /home/user/certs/mysite.com.fullchain.pem;
    ssl_certificate_key /home/user/certs/mysite.com.privkey.key;

    # Proxy conf
    location / {
        proxy_pass              https://apimbo;
        proxy_set_header        X-Forwarded-Host   $host;
        proxy_set_header        X-Forwarded-Server $host;
        proxy_set_header        X-Forwarded-For    $proxy_add_x_forwarded_for;
        proxy_set_header        Host               $http_host;
        proxy_connect_timeout   90s;
        proxy_send_timeout      90s;
        proxy_read_timeout      90s;
    }

}

reverse apim-gw vhost :

upstream apim {

    server 192.168.0.5:8243;
    keepalive 32;

}

server {

    # Listen
    server_name apim.mysite.com;
    listen      443      ssl http2;
    listen      [::]:443 ssl http2;

    # Certs
    ssl_certificate     /home/user/certs/mysite.com.fullchain.pem;
    ssl_certificate_key /home/user/certs/mysite.com.privkey.key;

    # Proxy conf
    location / {
        proxy_pass              https://apim;
        proxy_set_header        X-Forwarded-Host   $host;
        proxy_set_header        X-Forwarded-Server $host;
        proxy_set_header        X-Forwarded-For    $proxy_add_x_forwarded_for;
        proxy_set_header        Host               $http_host;
        proxy_set_header        Connection         "";
        proxy_connect_timeout   90s;
        proxy_send_timeout      90s;
        proxy_read_timeout      90s;
    }

}

apim-bo /etc/hosts :

127.0.1.1 apimbo.mysite.com
127.0.0.1 localhost
192.168.0.5 apim.mysite.com

apim-bo deployment.toml :

[server]
hostname = "apimbo.mysite.com"
base_path = "${carbon.protocol}://${carbon.host}:${carbon.management.port}"
server_role = "control-plane"

[super_admin]
username = "admin"
password = "adminpass"
create_admin_account = true

[user_store]
type = "database_unique_id"

[database.apim_db]
type = "mysql"
hostname = "192.168.0.4"
name = "apimdb"
port = "3306"
username = "apimdbuser"
password = "apimdbpass"
driver = "com.mysql.cj.jdbc.Driver"

[database.shared_db]
type = "mysql"
hostname = "192.168.0.4"
name = "apimshareddb"
port = "3306"
username = "apimshareddbuser"
password = "apimshareddbpass"
driver = "com.mysql.cj.jdbc.Driver"

[keystore.tls]
file_name = "wso2carbon.jks"
type = "PKCS12"
password = "wso2"
alias = "apim"
key_password = "wso2"

[keystore.primary]
file_name = "wso2carbon.jks"
type = "{{ wso2_keystore_type }}"
password = "{{ wso2_keystore_new_password }}"
alias = "{{ wso2_keystore_new_alias }}"
key_password = "{{ wso2_keystore_new_password }}"

[keystore.internal]
file_name = "wso2carbon.jks"
type = "PKCS12"
password = "{{ wso2_keystore_new_password }}"
alias = "{{ wso2_keystore_new_alias }}"
key_password = "{{ wso2_keystore_new_password }}"

[truststore]
file_name = "client-truststore.jks"
type = "PKCS12"
password = "wso2"
alias = "symmetric.key.value"
algorithm = "AES"

# Custom : apim gateway env
[[apim.gateway.environment]]
name = "Default"
type = "hybrid"
display_in_api_console = true
description = "This is a hybrid gateway that handles both production and sandbox token traffic."
show_as_token_endpoint_url = true
service_url = "https://apim.mysite.com/services/"
username= "${admin.username}"
password= "${admin.password}"
ws_endpoint = "ws://apim.mysite.com:9099"
wss_endpoint = "wss://apim.mysite.com:8099"
http_endpoint = "http://apim.mysite.com"
https_endpoint = "https://apim.mysite.com"

[apim.cache.tags]
expiry_time = "2m"

[apim.cache.jwt_claim]
enable = true

[apim.cache.gateway_token]
enable = true

[apim.cache.resource]
enable = true

[apim.cache.km_token]
enable = true

[apim.cache.scopes]
enable = true

[apim.oauth_config]
revoke_endpoint = "https://apimbo.mysite.com/oauth2/revoke"

[apim.devportal]
url = "https://apimbo.mysite.com/devportal"
api_key_alias = "apim"
display_multiple_versions = true

[apim.cors]
allow_origins = "*"
allow_methods = ["GET","PUT","POST","DELETE","PATCH","OPTIONS"]
allow_headers = ["authorization","Access-Control-Allow-Origin","Content-Type","SOAPAction", "apikey","Internal-Key"]
allow_credentials = false

[apim.workflow]
token_endpoint = "https://apimbo.mysite.com/oauth2/token"

[apim.notification]
hostname = "apimbo.mysite.com"

[[event_handler]]
name="userPostSelfRegistration"
subscriptions=["POST_ADD_USER"]

[service_provider]
sp_name_regex = "^[\sa-zA-Z0-9._-]*$"

[database.local]
url = "jdbc:h2:./repository/database/WSO2CARBON_DB;DB_CLOSE_ON_EXIT=FALSE"

[[event_listener]]
id = "token_revocation"
type = "org.wso2.carbon.identity.core.handler.AbstractIdentityHandler"
name = "org.wso2.is.notification.ApimOauthEventInterceptor"
order = 1
[event_listener.properties]
notification_endpoint = "https://localhost:${mgt.transport.https.port}/internal/data/v1/notify"
username = "${admin.username}"
password = "${admin.password}"
'header.X-WSO2-KEY-MANAGER' = "default"

[transport.https.properties]
proxyPort = 443

apim-gw /etc/hosts :

127.0.1.1 apim.mysite.com
127.0.0.1 localhost
192.168.0.6 apimbo.mysite.com

apim-gw deployment.toml :

[server]
hostname = "apim.mysite.com"
server_role = "gateway-worker"

[super_admin]
username = "admin"
password = "adminpass"
create_admin_account = true

[user_store]
type = "database_unique_id"

[database.shared_db]
type = "mysql"
hostname = "192.168.0.4"
name = "apimshareddb"
port = "3306"
username = "apimshareddbname"
password = "apimshareddbpass"
driver = "com.mysql.cj.jdbc.Driver"

[keystore.tls]
file_name = "wso2carbon.jks"
type = "PKCS12"
password = "wso2"
alias = "apim"
key_password = "wso2"

[keystore.primary]
file_name = "wso2carbon.jks"
type = "PKCS12"
password = "wso2"
alias = "apim"
key_password = "wso2"

[keystore.internal]
file_name = "wso2carbon.jks"
type = "PKCS12"
password = "wso2"
alias = "apim"
key_password = "wso2"

[truststore]
file_name = "client-truststore.jks"
type = "PKCS12"
password = "wso2"
alias = "symmetric.key.value"
algorithm = "AES"

[apim.key_manager]
service_url = "https://apimbo.mysite.com:9443${carbon.context}/services/"
username= "$ref{super_admin.username}"
password= "$ref{super_admin.password}"

[apim.jwt]
enable = true
encoding = "base64"
claim_dialect = "http://wso2.org/claims"
header = "X-JWT-Assertion"
signing_algorithm = "SHA256withRSA"

[apim.sync_runtime_artifacts.gateway]
gateway_labels = ["Default"]

[apim.throttling]
service_url = "https://apimbo.mysite.com:9443/services/"
username = "$ref{super_admin.username}"
password = "$ref{super_admin.password}"
enable_unlimited_tier = true
enable_header_based_throttling = false
enable_jwt_claim_based_throttling = false
enable_query_param_based_throttling = false
throttle_decision_endpoints = ["tcp://apimbo.mysite.com:5672","tcp://apimbo.mysite.com:5672"]

[[apim.throttling.url_group]]
traffic_manager_urls=["tcp://apimbo.mysite.com:9611"]
traffic_manager_auth_urls=["ssl://apimbo.mysite.com:9711"]

[apim.analytics]
enable = false
config_endpoint = "https://analytics-event-auth.choreo.dev/auth/v1"
auth_token = "ffffffff-ffff-ffff-ffff-ffffffffffff"

[apim.cache.jwt_claim]
enable = true
expiry_time = 900

[apim.cache.gateway_token]
enable = true
expiry_time = 15

[apim.cache.resource]
enable = true

[apim.cache.km_token]
enable = true

[apim.oauth_config]
remove_outbound_auth_header = true
auth_header = "Authorization"

[apim.devportal]
api_key_alias = "apim"

[apim.cors]
allow_origins = "*"
allow_methods = ["GET","PUT","POST","DELETE","PATCH","OPTIONS"]
allow_headers = ["authorization","Access-Control-Allow-Origin","Content-Type","SOAPAction", "apikey"]
allow_credentials = false

[transport.http]
properties.port = 9763
properties.proxyPort = 80

[transport.https]
properties.port = 9443
properties.proxyPort = 443

[passthru_http]
worker_thread_keepalive_sec = 60
worker_pool_queue_length = -1
core_worker_pool_size = 400
max_worker_pool_size = 400
io_buffer_size = 16384
http.socket.timeout = 180000

[message_builder]
json = "org.apache.synapse.commons.json.JsonStreamBuilder"
Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    Ok I managed to configure the control-plane as the key manager, so my errors are finally gone.

    Basically the gateway tried to claim /oauth2/jwks on the gateway side instead of calling it through the control-plane. Special thanks to you @ycr you helped me a lot.

    Nothing worked on deployment.toml (it looks like a bug to me), but I succeeded using the admin panel :

    I restarted both gw and cp to make sure that the value was not overwritten after a restart, and it's not.


  2. 0

    Add the following configs to your control plane deployment.toml

    If you have a single server.

    [apim.key_manager]
service_url = "https://localhost:9443/services/"
username = "admin"
password = "admin"

    If you have a HA setup

    [apim.key_manager]
service_url = "https://CP_LB_HOST/services/"
username = "admin"
password = "admin"

    Update

    Add the following to change the JWKS endpoint.

    [[apim.jwt.issuer]]
name = "https://localhost:9443/oauth2/token"
jwks.url = "https://CP_HOST:9443/oauth2/jwks"
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search