Amazon Web Services Question posted in
The official Amazon Web Services documentation can be found here.

Amazon web services – cant set AWS MSK kafka advertised.listeners , can anyone please explain what to do next? how to it?

For broker 1


bin/kafka-configs.sh --bootstrap-server b-1.kaffka.6r40l0.c4.kafka.eu-central-1.amazonaws.com:9094 
--entity-type brokers --entity-name 1 --alter --command-config client.properties 
--add-config advertised.listeners=[INTERNAL://b-1.kaffka.6r40l0.c4.kafka.eu-central-1.amazonaws.com:9096,EXTERNAL://MSK-kafka-Loadbalancer-25831a7b958a5031.elb.eu-central-1.amazonaws.com:9094]

For broker 2

bin/kafka-configs.sh --bootstrap-server b-2.kaffka.6r40l0.c4.kafka.eu-central-1.amazonaws.com:9094 
--entity-type brokers --entity-name 2 --alter --command-config client.properties 
--add-config advertised.listeners=[INTERNAL://b-2.kaffka.6r40l0.c4.kafka.eu-central-1.amazonaws.com:9096,EXTERNAL://MSK-kafka-Loadbalancer-25831a7b958a5031.elb.eu-central-1.amazonaws.com:9095],

all works fine here

bin/kafka-topics.sh --list --bootstrap-server b-1.kaffka.6r40l0.c4.kafka.eu-central-1.amazonaws.com:9096 --command-config client.
properties
__amazon_msk_canary
__consumer_offsets
test-topic-1

this is were im facing issue

[ec2-user@ip-10-1-1-98 kafka_2.13-3.5.1]$   bin/kafka-configs.sh --bootstrap-server b-1.kaffka.6r40l0.c4.kafka.eu-central-1.amazonaws.com:9096 
--entity-type brokers --entity-name 1 --alter --command-config client.properties 
--add-config advertised.listeners=[INTERNAL://b-1.kaffka.6r40l0.c4.kafka.eu-central-1.amazonaws.com:9096,EXTERNAL://MSK-kafka-Loadbalancer-25831a7b958a5031.elb.eu-central-1.amazonaws.com:9094]
Error while executing config command with args '--bootstrap-server b-1.kaffka.6r40l0.c4.kafka.eu-central-1.amazonaws.com:9096 --entity-type brokers --entity-name 1 --alter --command-config client.properties --add-config advertised.listeners=[INTERNAL://b-1.kaffka.6r40l0.c4.kafka.eu-central-1.amazonaws.com:9096,EXTERNAL://MSK-kafka-Loadbalancer-25831a7b958a5031.elb.eu-central-1.amazonaws.com:9094]'
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Error creating broker listeners from 'INTERNAL://b-1.kaffka.6r40l0.c4.kafka.eu-central-1.amazonaws.com:9096,EXTERNAL://MSK-kafka-Loadbalancer-25831a7b958a5031.elb.eu-central-1.amazonaws.com:9094': No security protocol defined for listener INTERNAL
        at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
        at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2028)
        at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:180)
        at kafka.admin.ConfigCommand$.alterConfig(ConfigCommand.scala:380)
        at kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:328)
        at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:97)
        at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
Caused by: org.apache.kafka.common.errors.InvalidRequestException: Error creating broker listeners from 'INTERNAL://b-1.kaffka.6r40l0.c4.kafka.eu-central-1.amazonaws.com:9096,EXTERNAL://MSK-kafka-Loadbalancer-25831a7b958a5031.elb.eu-central-1.amazonaws.com:9094': No security protocol defined for listener INTERNAL

this is were im facing issue

[ec2-user@ip-10-1-1-98 kafka_2.13-3.5.1]$ bin/kafka-configs.sh --bootstrap-server b-1.kaffka.6r40l0.c4.kafka.eu-central-1.amazonaws.com:9096 
--entity-type brokers --entity-name 1 --alter --command-config client.properties 
--add-config advertised.listeners=[INTERNAL://b-1.kaffka.6r40l0.c4.kafka.eu-central-1.amazonaws.com:9096,EXTERNAL://MSK-kafka-Loadbalancer-25831a7b958a5031.elb.eu-central-1.amazonaws.com:9094],listener.security.protocol.map=INTERNAL:SSL,EXTERNAL:SSL
requirement failed: Invalid entity config: all configs to be added must be in the format "key=val".
[ec2-user@ip-10-1-1-98 kafka_2.13-3.5.1]$

im trying to set my NLB DNS for my aws msk kafka brokers

Tags:

2

Answers


  1. 0

    No security protocol defined for listener INTERNAL

    Sounds like you want to define listener.security.protocol.map before trying to set up the listeners, as it’s own --add-config argument rather than trying multiple at a time.

    Internal & public access ports are already well defined, though https://docs.aws.amazon.com/msk/latest/developerguide/port-info.html

    Kafka doesn’t really need any load balancer, since clients must communicate with individual brokers, eventually. It would only be useful for a single bootstrap address (but then causes a single point of failure)

    Login or Signup to reply.
  2. 0

    When you want to update advertised listeners, you should prepare the new value following these steps:

    Get an existing value for advertised listeners from each broker as following:

    kafka/bin/kafka-configs.sh --bootstrap-server $b 
--command-config config/client_iam.properties  
--entity-type brokers --entity-name 1 --describe 
--all 
|grep advertised.listeners

    The output will look like this (formatted to multi-line for readability):

    advertised.listeners=
CLIENT://b-1.my-msk.a1234a.c8.kafka.us-west-2.amazonaws.com:9096,
CLIENT_IAM://b-1.my-msk.a1234a.c8.kafka.us-west-2.amazonaws.com:9098,
CLIENT_SECURE://b-1.my-msk.a1234a.c8.kafka.us-west-2.amazonaws.com:9094,
REPLICATION://b-1-internal.my-msk.a1234a.c8.kafka.us-west-2.amazonaws.com:9093,
REPLICATION_SECURE://b-1-internal.my-msk.a1234a.c8.kafka.us-west-2.amazonaws.com:9095

    As you can see, there are couple of protocols: CLIENT (SASL-SCRAM), CLIENT_IAM (IAM), CLIENT_SECURE (TLS). You can change only one of those. Not sure what happens when you change internal protocols like REPLICATION or REPLICATION_SECURE, but pretty sure you will have you cluster unstable (at very least).

    Now, you need to modify that single-line value (not formatted with new-lines).
    You have to modify only the PROTOCOL you want to change. I would recommend connecting to the admin API with different protocol that you would like to change.
    For instance, if you want to change TLS (CLIENT_SECURE on 9094), use IAM authentication (port 9098).

    So, in your case the value would look like following:

    kafka/bin/kafka-configs.sh --bootstrap-server b-1.my-msk.a1234a.c8.kafka.us-west-2.amazonaws.com:9098 
--entity-type brokers --entity-name 1 --alter 
--command-config client_iam.properties 
--add-config 
advertised.listeners=[CLIENT://b-1.my-msk.a1234a.c8.kafka.us-west-2.amazonaws.com:9096,
CLIENT_IAM://b-1.my-msk.a1234a.c8.kafka.us-west-2.amazonaws.com:9098,
CLIENT_SECURE://MSK-kafka-Loadbalancer-25831a7b958a5031.elb.eu-central-1.amazonaws.com:9094,
REPLICATION://b-1-internal.my-msk.a1234a.c8.kafka.us-west-2.amazonaws.com:9093,
REPLICATION_SECURE://b-1-internal.my-msk.a1234a.c8.kafka.us-west-2.amazonaws.com:9095]

    Do the same for the rest of the brokers, changing their instance IDs in the command you are executing and values.

    Note, after you change advertised listeners, brokers won’t start listening on other ports, they will only replace metadata during fetch response to your clients, so your clients will think the brokers’ hosts are MSK-kafka-Loadbalancer...

    At this point, you need already have ELB with target groups to original host and port (i.e. b-1.my-msk.a1234a.c8.kafka.us-west-2.amazonaws.com:9094), and in the client config for bootstrap servers you will need to use MSK-kafka-Loadbalancer-25831a7b958a5031.elb.eu-central-1.amazonaws.com:9094.

    Another note,
    Since you are changing domain name, you will break end-to-end TLS, and handshake won’t be successful. You will need to terminate TLS at NLB.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search