Question 190406 in Amazon Web Sevices Question posted in
The official Amazon Web Services documentation can be found here.

Amazon web services – AWS MSK kafka Not authorized to access topics

My Kafka cluster is IAM auth enabled. I am successfully able to produce and consume messages from topic test-topic2 by assuming the correct IAM role if the policy is as follows;

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowKafkaTopicWrite",
            "Effect": "Allow",
            "Action": [
                "kafka:*",
                "kafka-cluster:*"
            ],
            "Resource": "*"
        }
    ]
}

But now I want to narrow down the policy to a specific cluster, so I change it to following;

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowKafkaTopicWrite",
            "Effect": "Allow",
            "Action": [
                "kafka:*",
                "kafka-cluster:*"
            ],
            "Resource": [
                "arn:aws:kafka:eu-west-1:123456789:cluster/test-cluster/9f4ea0a3-75bc-4ff9-a971-73efa2ef73c9-9",
                "arn:aws:kafka:eu-west-1:123456789:cluster/test-cluster/9f4ea0a3-75bc-4ff9-a971-73efa2ef73c9-9/*",
                "arn:aws:kafka:eu-west-1:123456789:cluster/test-cluster/9f4ea0a3-75bc-4ff9-a971-73efa2ef73c9-9/topic/test-topic2"
            ]
        }
    ]
}

I get following error on Producer side;

[2023-06-15 15:28:42,476] ERROR Error when sending message to topic test-topic2 with key: null, value: 1 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [test-topic2]

And on consumer;

[2023-06-15 15:28:00,208] WARN [Consumer clientId=console-consumer, groupId=console-consumer-46486] Error while fetching metadata with correlation id 3 : {test-topic2=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
[2023-06-15 15:28:00,210] ERROR [Consumer clientId=console-consumer, groupId=console-consumer-46486] Topic authorization failed for topics [test-topic2] (org.apache.kafka.clients.Metadata)
[2023-06-15 15:28:00,211] ERROR Error processing message, terminating consumer process:  (kafka.tools.ConsoleConsumer$)
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [test-topic2]

What am I missing here ?

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    After manually testing policies, turns out I was missing grants to group as well as transactional-id. And ofcourse the topic grant was incorrect (as pointed out by @MrocKK). Here is the minimal policy that works (read/Write on/to test-topic2)

            {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:DescribeTopicDynamicConfiguration",
                "kafka-cluster:AlterGroup",
                "kafka-cluster:WriteDataIdempotently",
                "kafka-cluster:DescribeCluster",
                "kafka-cluster:ReadData",
                "kafka-cluster:DescribeTopic",
                "kafka-cluster:DescribeTransactionalId",
                "kafka-cluster:DescribeGroup",
                "kafka-cluster:DescribeClusterDynamicConfiguration",
                "kafka-cluster:Connect",
                "kafka-cluster:WriteData"
            ],
            "Resource": [
                "arn:aws:kafka:eu-west-1:123456789:cluster/test-cluster/9f4ea0a3-75bc-4ff9-a971-73efa2ef73c9-9",
                "arn:aws:kafka:eu-west-1:123456789:topic/test-cluster/9f4ea0a3-75bc-4ff9-a971-73efa2ef73c9-9/test-topic2",
                "arn:aws:kafka:eu-west-1:123456789:group/test-cluster/9f4ea0a3-75bc-4ff9-a971-73efa2ef73c9-9/*",
                "arn:aws:kafka:eu-west-1:123456789:transactional-id/test-cluster/9f4ea0a3-75bc-4ff9-a971-73efa2ef73c9-9/*"
            ]
        }

    And to make it read-only, simple remove WriteDataIdempotently and WriteData from Actions.


  2. 0

    topic arn in your IAM policy seems incorrect, as per the documentation – https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html#msk-iam-resources
    Topic – arn:aws:kafka:region:account-id:topic/cluster-name/cluster-uuid/topic-name

    so the topic arn in this case should be arn:aws:kafka:eu-west-1:123456789:topic/test-cluster/9f4ea0a3-75bc-4ff9-a971-73efa2ef73c9-9/test-topic2

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search