I am using this Github repo and folder path I found: https://github.com/entechlog/kafka-examples/tree/master/kafka-connect-standalone
The issue I am having is that, as a matter of access control, I must specify my group ID by adding a prefix to it, let’s call it abc-
. When I build this Docker image, I check my logs and I can see that the group ID ends up being connect-bq-sink-connector
, which I am assuming is a concatenation of the word connect-
along with the variable CONNECTOR_NAME
seen in the docker-compose file. When I change the connector name variable, my group ID also changes (but the connect-
prefix always remains). You will also see a variable called CONNECT_GROUP_ID
in the docker-compose file. This variable appears to have absolutely no effect on the Kafka connect instance. The Docker logs give this (in this order):
WARN The configuration 'group.id' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:380)
and then later…
…
group.id = connect-bq-sink-connector
The final error, which is mostly unimportant as I know it is due to lack of permissions, is simply:
[2021-12-03 22:37:32,775] INFO [bq-sink-connector|task-0] [Consumer clientId=connector-consumer-bq-sink-connector-0, groupId=connect-bq-sink-connector] FindCoordinator request hit fatal exception (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:260)
org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access Group: connect-bq-sink-connector
and so two things: I cannot figure out why the word connect-
is being prefixed every time to the resulting group ID, and also why the variable CONNECT_GROUP_ID seems to not function at all.
I thought maybe the image is forcibly hard coding connect-
in front. I tried running this by changing line 50 in the docker compose file from image: entechlog/cp-kafka-connect${CONFLUENT_VERSION_TAG}
to image: confluentinc/cp-kafka:5.0.0
and it changed nothing.
What is causing this prefix to be added, and how can I override this? I can keep the word connect
there if necessary, but I simply need to add my prefix abc-
in front of it.
3
Answers
If you want to change connect group id, add environment variable name
CONNECTOR_
properties section under the servicekafka-connect
and set a value you want.The github example starting steps as follows.
docker/Dockerfile
, a startup command is/etc/confluent/docker/run
and you cant find the file indocker/include/etc/confluent/docker
.configure
andlaunch
in thedocker/include/etc/confluent/docker/run
file.docker/include/etc/confluent/docker/configure
, Check mandatory environment variables such asCONNECT_BOOTSTRAP_SERVERS
,CONNECT_KEY_CONVERTER
,CONNECT_VALUE_CONVERTER
… are set, and call templating function withkafka-connect-standalone.properties.template
andkafka-connect.properties.template
.So if there is a configuration that you want to add to the
kafka-connect-standalone.properties
file, you must specify an environment variable starting withCONNECTOR_
.You can find all configuration for kafka connect in the following link.
The Sink Connector consumer group id prefix cannot be changed
https://issues.apache.org/jira/browse/KAFKA-4400
CONNECT_GROUP_ID
is only applicable for Kafka Connect distributed mode clustering, not consumers, which the Confluent images use. I don’t see any benefit of using the linked image that runs standalone since that would fail to be fault tolerant or scalable (the Connect configs and status are stored in an ephemeral filesystem rather than back in Kafka)Currently, there is no way to change the group id for the kafka sink connector.
consumer group id
is hardcoded and prefixed withconnect-
.Ref. https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/util/SinkUtils.java