KAFKA and SSL : java.lang.OutOfMemoryError: Java heap space when using kafka-topics command on KAFKA SSL cluster – CentOS

this is my first post on Stackoverflow, i hope i didnt choose the wrong section.

Context :

Kafka HEAP size is configured on following file :

/etc/systemd/system/kafka.service

With following parameter :

Environment="KAFKA_HEAP_OPTS=-Xms6g -Xmx6g"

OS is “CentOS Linux release 7.7.1908”.

Kafka is “confluent-kafka-2.12-5.3.1-1.noarch”, installed from the following repository :

# Confluent REPO
[Confluent.dist]
name=Confluent repository (dist)
baseurl=http://packages.confluent.io/rpm/5.3/7
gpgcheck=1
gpgkey=http://packages.confluent.io/rpm/5.3/archive.key
enabled=1

[Confluent]
name=Confluent repository
baseurl=http://packages.confluent.io/rpm/5.3
gpgcheck=1
gpgkey=http://packages.confluent.io/rpm/5.3/archive.key
enabled=1

I activated SSL on a 3-machine KAFKA cluster few days ago, and suddently, the following command stopped working :

kafka-topics --bootstrap-server <the.fqdn.of.server>:9093 --describe --topic <TOPIC-NAME>

Which return me the following error :

[2019-10-03 11:38:52,790] ERROR Uncaught exception in thread 'kafka-admin-client-thread | adminclient-1':(org.apache.kafka.common.utils.KafkaThread) 
java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
    at org.apache.kafka.common.memory.MemoryPool$1.tryAllocate(MemoryPool.java:30)
    at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:112)
    at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:424)
    at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:385)
    at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:651)
    at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:572)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:483)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:539)
    at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1152)
    at java.lang.Thread.run(Thread.java:748)

On the server’s log, the following line appears when i try to request it via “kafka-topics” :

/var/log/kafka/server.log :
[2019-10-03 11:41:11,913] INFO [SocketServer brokerId=<ID>] Failed authentication with /<ip.of.the.server> (SSL handshake failed) (org.apache.kafka.common.network.Selector)

I was able to use this command properly BEFORE implementing SSL on the cluster. Here is the configuration i’m using.
All functionnality work properly (consumers, producers…) except “kafka-topics” :

# SSL Configuration
ssl.truststore.location=<truststore-path>
ssl.truststore.password=<truststore-password>
ssl.keystore.type=<keystore-type>
ssl.keystore.location=<keystore-path>
ssl.keystore.password=<keystore-password>

# Enable SSL between brokers
security.inter.broker.protocol=SSL

# Listeners
listeners=SSL://<fqdn.of.the.server>:9093
advertised.listeners=SSL://<fqdn.of.the.server>:9093

There is no problem with the certificate (which is signed by internal CA, internal CA which i added to the truststore specified on the configuration). OpenSSL show no errors :

openssl s_client -connect <fqdn.of.the.server>:9093 -tls1
>> Verify return code: 0 (ok)

The following command is working pretty well with SSL, thanks to parameter “-consumer.config client-ssl.properties”

kafka-console-consumer --bootstrap-server <fqdn.of.the.server>:9093 --topic <TOPIC-NAME> -consumer.config client-ssl.properties

“client-ssl.properties” content is :

security.protocol=SSL
ssl.truststore.location=<truststore-path>
ssl.truststore.password=<truststore-password>

Right now, i’m forced to use “–zookeeper”, which according to the documentation, is deprecated :

--zookeeper <String: hosts>              DEPRECATED, The connection string for  
                                       the zookeeper connection in the form 
                                       host:port. Multiple hosts can be     
                                       given to allow fail-over. 

And of course, it’s working pretty well :

kafka-topics --zookeeper <fqdn.of.the.server>:2181 --describe --topic <TOPIC-NAME>
Topic:<TOPIC-NAME>  PartitionCount:3    ReplicationFactor:2 
Configs:
Topic: <TOPIC-NAME> Partition: 0    Leader: <ID-3>      Replicas: <ID-3>,<ID-1> Tsr: <ID-1>,<ID-3>
Topic: <TOPIC-NAME> Partition: 1    Leader: <ID-1>      Replicas: <ID-1>,<ID-2> Isr: <ID-2>,<ID-1>
Topic: <TOPIC-NAME> Partition: 2    Leader: <ID-2>      Replicas: <ID-2>,<ID-3> Isr: <ID-2>,<ID-3>

So, my question is : why am i unable to use “–bootstrap-server” atm ? Because of the “zookeeper” deprecation, i’m worried about not to be able to consult my topics, and their details…

I believe that kafka-topics needs the same option than kafka-console-consumer, aka “-consumer.config”…

Ask if any additionnal precision needed.

Thanks a lot, hope my question is clear and readable.

Blyyyn