AWS MSK IAM: Access denied

[devdavidkarlsson]

Describe the bug

AWS MSK ECS with IAM, I am experiencing something similar to this. The logs show:

2021-11-16T11:46:58.120+01:00 | 10:46:58.118 [main] INFO com.provectus.kafka.ui.KafkaUiApplication - Started KafkaUiApplication in 34.704 seconds (JVM running for 50.263)
-- | --
  | 2021-11-16T11:47:10.979+01:00 | 10:47:10.976 [kafka-admin-client-thread \| adminclient-1] ERROR com.provectus.kafka.ui.service.KafkaService - Failed to collect cluster air-ue2-test-msk-test-704e info
  | 2021-11-16T11:47:10.979+01:00 | org.apache.kafka.common.errors.SaslAuthenticationException: [7d0ab893-9b16-4371-a8b2-1ae9b5ce73a5]: Access denied
  | 2021-11-16T11:47:24.627+01:00 | 10:47:24.621 [parallel-1] DEBUG com.provectus.kafka.ui.service.MetricsUpdateService - Start getting metrics for kafkaCluster: air-ue2-test-msk-test-704e
  | 2021-11-16T11:47:25.148+01:00 | 10:47:25.148 [kafka-admin-client-thread \| adminclient-2] ERROR com.provectus.kafka.ui.service.KafkaService - Failed to collect cluster air-ue2-test-msk-test-704e info
  | 2021-11-16T11:47:25.148+01:00 | org.apache.kafka.common.errors.SaslAuthenticationException: [783b3500-7435-4f17-a929-c6ea028bb246]: Access denied
  | 2021-11-16T11:47:54.621+01:00 | 10:47:54.621 [parallel-1] DEBUG com.provectus.kafka.ui.service.MetricsUpdateService - Start getting metrics for kafkaCluster: air-ue2-test-msk-test-704e
  | 2021-11-16T11:47:55.042+01:00 | 10:47:55.042 [kafka-admin-client-thread \| adminclient-3] ERROR com.provectus.kafka.ui.service.KafkaService - Failed to collect cluster air-ue2-test-msk-test-704e info
  | 2021-11-16T11:47:55.042+01:00 | org.apache.kafka.common.errors.SaslAuthenticationException: [55b0c135-b3a8-43fc-99a8-388fa7a0ddea]: Too many connects

Set up

{
    "taskDefinitionArn": "arn:aws:ecs:us-east-1:xxxxx:task-definition/kafkaui:10",
    "containerDefinitions": [
        {
            "name": "kafkaui",
            "image": "provectuslabs/kafka-ui:latest",
            "cpu": 256,
            "memory": 512,
            "memoryReservation": 128,
            "portMappings": [
                {
                    "containerPort": 8080,
                    "hostPort": 8080,
                    "protocol": "tcp"
                }
            ],
            "essential": true,
            "environment": [
                {
                    "name": "KAFKA_CLUSTERS_0_NAME",
                    "value": "air-ue2-test-msk-test-704e"
                },
                {
                    "name": "KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL",
                    "value": "SASL_SSL"
                },
                {
                    "name": "LOGGING_LEVEL_COM_PROVECTUS",
                    "value": "debug"
                },
                {
                    "name": "KAFKA_CLUSTERS_0_PROPERTIES_SASL_CLIENT_CALLBACK_HANDLER_CLASS",
                    "value": "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
                },
                {
                    "name": "KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG",
                    "value": "software.amazon.msk.auth.iam.IAMLoginModule required awsDebugCreds=true;"
                },
                {
                    "name": "KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM",
                    "value": "AWS_MSK_IAM"
                },
                {
                    "name": "KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS",
                    "value": "b-1.air-ue2-test-msk-test.xxxxx.c20.kafka.us-east-1.amazonaws.com:9098,b-2.air-ue2-test-msk-test.xxxxx.c20.kafka.us-east-1.amazonaws.com:9098,b-3.air-ue2-test-msk-test.xxxxx.c20.kafka.us-east-1.amazonaws.com:9098"
                }
            ],
            "mountPoints": [],
            "volumesFrom": [],
            "startTimeout": 30,
            "stopTimeout": 30,
            "readonlyRootFilesystem": false,
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "kafkaui",
                    "awslogs-region": "us-east-1",
                    "awslogs-stream-prefix": "ecs"
                },
                "secretOptions": []
            }
        }
    ],
    "family": "kafkaui",
    "taskRoleArn": "arn:aws:iam::xxxxx:role/kafkaui-ecs_task_execution",
    "executionRoleArn": "arn:aws:iam::xxxxx:role/kafkaui-ecs_task_execution",
    "networkMode": "awsvpc",
    "revision": 10,
    "volumes": [],
    "status": "ACTIVE",
    "requiresAttributes": [
        {
            "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
        },
        {
            "name": "ecs.capability.execution-role-awslogs"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.21"
        },
        {
            "name": "com.amazonaws.ecs.capability.task-iam-role"
        },
        {
            "name": "ecs.capability.container-ordering"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
        },
        {
            "name": "ecs.capability.task-eni"
        }
    ],
    "placementConstraints": [],
    "compatibilities": [
        "EC2",
        "FARGATE"
    ],
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "cpu": "256",
    "memory": "512",
    "registeredAt": "2021-11-16T10:01:05.437Z",
    "registeredBy": "arn:aws:sts::xxxxx",
    "tags": [
        {
            "key": "Name",
            "value": "kafkaui"
        }
    ]
}

To test this out I have set a quite liberal execution policy on the task: "arn:aws:iam::aws:policy/AmazonMSKFullAccess"

Steps to Reproduce
Steps to reproduce the behavior:

1. Setup MSK with:

number_of_broker_nodes = 3
broker_instance_type = "kafka.t3.small"

2. setup kafka ui in ecs, allow the security groups to access kafka.
3. the kafka ui logs show failing to authenticate: too many connnects.

Expected behavior

Successful IAM authentication

Screenshots

Additional context

Would it be possible to set: reconnect.backoff.ms somewhere for kafkaui?

I can successfully connect and read, if I turn of IAM, and remove the related task config for MSK_IAM, that is remove:

                {
                    "name": "KAFKA_CLUSTERS_0_PROPERTIES_SASL_CLIENT_CALLBACK_HANDLER_CLASS",
                    "value": "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
                },
                {
                    "name": "KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG",
                    "value": "software.amazon.msk.auth.iam.IAMLoginModule required awsDebugCreds=true;"
                },
                {
                    "name": "KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM",
                    "value": "AWS_MSK_IAM"
                },

[github-actions[bot]]

Hello there devdavidkarlsson! 👋

Thank you and congratulations 🎉 for opening your very first issue in this project! 💖

In case you want to claim this issue, please comment down below! We will try to get back to you as soon as we can. 👀

[devdavidkarlsson]

fix:

• update instance size from kafka.t3.small --> kafka.m5.large
• add permission to task execution role: kafka-cluster:Connect

[yogenderPalChandra]

In addition to the above answer by devdavidkarlsson. The following policy has to be applied to the EMR_EC2_Default_Role (or the role EMR master node is using):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:Connect",
                "kafka-cluster:AlterCluster",
                "kafka-cluster:DescribeCluster"
            ],
            "Resource": [
                "arn:aws:kafka:eu-central-1:73827389XXX:cluster/ClusterName/b3ec24e2-b893-4372-ba17-96fff3176a7f-5"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:*Topic*",
                "kafka-cluster:WriteData",
                "kafka-cluster:ReadData"
            ],
            "Resource": [
                "arn:aws:kafka:eu-central-1:37892738912XXX:topic/clusterName/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:AlterGroup",
                "kafka-cluster:DescribeGroup"
            ],
            "Resource": [
                "arn:aws:kafka:eu-central-1:636XXXX:group/ClusterName/*"
            ]
        }
    ]
}

Source:
https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html

[devdavidkarlsson]

(remaining issue: cant create topics)

[Haarolean]

@devdavidkarlsson hey, thanks for the info. I'll convert this to a discussion.
Regarding a problem with creating topics, could you please raise a separate issue? Thanks.

[mng1dev]

Hi,
In addition to what has already been said here, I wanted to add that today I struggled to connect kafka-ui to a MSK broker, and the problem was the client not having the proper authorization to perform a DESCRIBE_CONFIGS request.

According to msk documentation, the IAM policy should also contain the authorization to perform the kafka-cluster:DescribeClusterDynamicConfiguration operation on the cluster.

[Haarolean]

Thank you for the information :)