Question 18601 in Amazon Web Sevices Question posted in
The official Amazon Web Services documentation can be found here.

Authorization issue when performing operations on MSK cluster using SASL/SCRAM auth method – Amazon Web Sevices

I have configured the MSK cluster and allowed public access through SASL/SCRAM authentication method. Now I am facing the issue where I do not have the necessary permissions when using these credentials (specified in the Secrets Manager created with a custom key). The connecting client can perform certain operations (e.g. retrieve metadata) but fails to fetch or create topics or publish a new message to the existing topic. I am using Confluent as the library and here is a simplified example of the configuration that I am using (this is probably not relevant at all but is here to support an explanation of the issue).

BootstrapServers = Config.KafkaBootstrapServers,
SaslMechanism = SaslMechanism.ScramSha512, // only supported option by AWS
SecurityProtocol = SecurityProtocol.SaslSsl,
SaslUsername = Config.Username, // username from secrets manager
SaslPassword = Config.Password, // password from secrets manager
ClientId = Config.Client,
Acks = Acks.All

The error I get is Confluent.Kafka.Admin.CreateTopicsException: An error occurred creating topics: [topic]: [Authorization failed.]

How can I assign higher permissions? Since it is a managed Kafka service, there is no option to modify this on broker level directly. And since there is no user behind these credentials (since I am not using IAM auth method), I cannot assign a specific policy to it allowing certain operations like topic creation. What are the options here?

This page explains how ACLs are generally configured on Kafka but not on MSK. Am I missing something here?

Tags:

2

Answers


  1. 0

    In MSK with SASL/SCRAM, authorizations are performed using ACLs. However, note that MSK sets "allow.everyone.if.no.acl.found" to true by default

    From public docs:

    This means that with Amazon MSK clusters, if you don't explicitly set ACLs on a resource, all principals can access this resource. If you enable ACLs on a resource, only the authorized principals can access it.

    I suspect there is some existing ACL on these resources which is preventing you to produce/create topics. Would you be able to list ACLs for this cluster and verify this?

    Login or Signup to reply.
  2. 0

    Kafka ACLs commands are working perfectly fine with MSK to control authorization policies on a cluster.

    Define policies for the user that you are connecting with – Config.Username, with appropriate permissions.

    # allow topic creation
kafka-acls zookeeper.connect=z-1:2181 
--add --allow-principal User:THE_USER 
--operation DESCRIBE 
--operation CREATE 
--cluster


# allow read from and write to a topic
kafka-acls zookeeper.connect=z-1:2181 
--add --allow-principal User:THE_USER 
--operation read --operation write --topic some-kafka-topic
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search