Question 420713 in Amazon Web Sevices Question posted in
The official Amazon Web Services documentation can be found here.

Amazon web services – AWS Lambda code to connect with EKS cluster

I have a lambda code in python (v3.13) which is trying to connect to an AWS EKS cluster to run a job. The lambda and AWS EKS are in same VPC, subnets, and have same security groups so that there will not be any issue related to communication.

Lambda code in python3.13:

import os
import boto3
import kubernetes
from kubernetes import client, config
from botocore.exceptions import ClientError

def lambda_handler(event, context):
    # Step 1: Get the cluster name 
    cluster_name = "app1-eks-cluster"
    
    # Step 2: Retrieve the EKS cluster details to get the kubeconfig
    eks_client = boto3.client('eks')
    try:
        # Get EKS cluster information
        print("Getting cluster info...")
        cluster_info = eks_client.describe_cluster(name=cluster_name)
        cluster_endpoint = cluster_info['cluster']['endpoint']
        cluster_certificate = cluster_info['cluster']['certificateAuthority']['data']
        print("cluster_endpoint=" + cluster_endpoint)
    except ClientError as e:
        print(f"Error retrieving cluster details: {e}")
        raise e
    
    # Step 3: Create Kubernetes configuration object to connect to the EKS cluster
    try:
        kube_config = {
            'api_version': 'v1',
            'clusters': [{
                'cluster': {
                    'server': cluster_endpoint,
                    'certificate-authority-data': cluster_certificate
                },
                'name': cluster_name
            }],
            'contexts': [{
                'context': {
                    'cluster': cluster_name,
                    'user': cluster_name
                },
                'name': cluster_name
            }],
            'current-context': cluster_name,
            'kind': 'Config',
            'users': [{
                'name': cluster_name,
                'user': {
                    'exec': {
                        'apiVersion': 'client.authentication.k8s.io/v1beta1',
                        'command': 'aws',
                        'args': ['eks', 'get-token', '--cluster-name', cluster_name]
                    }
                }
            }]
        }
        
        # Set the kubeconfig
        config.load_kube_config_from_dict(kube_config)
        
    except Exception as e:
        print(f"Error setting up Kubernetes client: {e}")
        raise e
    
    # Step 4: Run a Kubernetes job (or other action) within the EKS cluster
    try:
        # Example: Create a simple Job in Kubernetes
        batch_v1 = client.BatchV1Api()
        job = client.V1Job(
            metadata=client.V1ObjectMeta(name="example-job"),
            spec=client.V1JobSpec(
                template=client.V1PodTemplateSpec(
                    spec=client.V1PodSpec(
                        containers=[client.V1Container(name="example-container", image="busybox", command=["/bin/sh", "-c", "echo Hello Kubernetes"])],
                        restart_policy="Never"
                    )
                ),
                backoff_limit=4
            )
        )
        
        # Submit the job to the Kubernetes cluster
        api_response = batch_v1.create_namespaced_job(
            body=job,
            namespace="f-design"
        )
        print(f"Job created. Status: {api_response.status}")
    
    except Exception as e:
        print(f"Error running job on EKS: {e}")
        raise e

    return {
        'statusCode': 200,
        'body': 'Job successfully submitted to the EKS cluster'
    }

Error:

Response:
{
  "errorMessage": "(403)nReason: ForbiddennHTTP response headers: HTTPHeaderDict({'Audit-Id': '21cfdf80-e0f7-45db-1234-65a3429a90ff', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'e60b20c4-5970-5678-1234-856e48a23731', 'X-Kubernetes-Pf-Prioritylevel-Uid': '4a2c47d9-bc5c-1234-5678-d36bb38878f0', 'Date': 'Fri, 27 Dec 2024 07:05:37 GMT', 'Content-Length': '305'})nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:anonymous\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"f-design\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}nn",
  "errorType": "ApiException",
  "requestId": "4f536051-0e16-4ddd-1234-5e6799767e78",
  "stackTrace": [
    "  File "/var/task/lambda_function.py", line 89, in lambda_handlern    raise en",
    "  File "/var/task/lambda_function.py", line 81, in lambda_handlern    api_response = batch_v1.create_namespaced_job(n",
    "  File "/var/task/kubernetes/client/api/batch_v1_api.py", line 210, in create_namespaced_jobn    return self.create_namespaced_job_with_http_info(namespace, body, **kwargs)  # noqa: E501n",
    "  File "/var/task/kubernetes/client/api/batch_v1_api.py", line 309, in create_namespaced_job_with_http_infon    return self.api_client.call_api(n",
    "  File "/var/task/kubernetes/client/api_client.py", line 348, in call_apin    return self.__call_api(resource_path, method,n",
    "  File "/var/task/kubernetes/client/api_client.py", line 180, in __call_apin    response_data = self.request(n",
    "  File "/var/task/kubernetes/client/api_client.py", line 391, in requestn    return self.rest_client.POST(url,n",
    "  File "/var/task/kubernetes/client/rest.py", line 279, in POSTn    return self.request("POST", url,n",
    "  File "/var/task/kubernetes/client/rest.py", line 238, in requestn    raise ApiException(http_resp=r)n"
  ]
}

Function Logs:
START RequestId: 4f536051-0e16-4ddd-1234-5e6799767e78 Version: $LATEST
Getting cluster info...
cluster_endpoint=https://12333333333333332.gr8.ap-south-1.eks.amazonaws.com
[ERROR] 2024-12-27T07:05:37.187Z    4f536051-0e16-4ddd-1234-5e6799767e78    [Errno 2] No such file or directory: 'aws'
Error running job on EKS: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '21cfdf80-e0f7-45db-1234-65a3429a90ff', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'e60b20c4-5970-5678-1234-856e48a23731', 'X-Kubernetes-Pf-Prioritylevel-Uid': '4a2c47d9-bc5c-1234-5678-d36bb38878f0', 'Date': 'Fri, 27 Dec 2024 07:05:37 GMT', 'Content-Length': '305'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User "system:anonymous" cannot create resource "jobs" in API group "batch" in the namespace "f-design"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}
LAMBDA_WARNING: Unhandled exception. The most likely cause is an issue in the function code. However, in rare cases, a Lambda runtime update can cause unexpected function behavior. For functions using managed runtimes, runtime updates can be triggered by a function change, or can be applied automatically. To determine if the runtime has been updated, check the runtime version in the INIT_START log entry. If this error correlates with a change in the runtime version, you may be able to mitigate this error by temporarily rolling back to the previous runtime version. For more information, see https://docs.aws.amazon.com/lambda/latest/dg/runtimes-update.html
[ERROR] ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '21cfdf80-e0f7-45db-1234-65a3429a90ff', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'e60b20c4-5970-5678-1234-856e48a23731', 'X-Kubernetes-Pf-Prioritylevel-Uid': '4a2c47d9-bc5c-1234-5678-d36bb38878f0', 'Date': 'Fri, 27 Dec 2024 07:05:37 GMT', 'Content-Length': '305'})

I have updated aws-auth-config and granted system:master permission to the lambda role for the lambda function that I have created. Something like this

mapRoles: |
- groups:
- system:masters
rolearn: arn:aws:iam::111122223333:role/lambdarole
username: lambdafunctionname

but still getting the same 403 forbidden error from eks.

Any help would be greatly appreciated!.

Tags:

2

Answers


  1. 0

    The primary error: [Errno 2] No such file or directory: ‘aws’
    This appears before the 403 error and this is a indication that the Lambda function can’t find the AWS CLI, which is needed for the token generation in your kubeconfig. This is why you’re being identified as system:anonymous

    Hence Suggest to removed the AWS CLI dependency *in the kube_config dictionary under the users section *

    Please Replace this with direct boto3 token generation

    ‘users’: [{
    ‘name’: cluster_name,
    ‘user’: {
    ‘exec’: { # This part tries to execute the AWS CLI
    ‘apiVersion’: ‘client.authentication.k8s.io/v1beta1’,
    ‘command’: ‘aws’, # <– This is trying to use the AWS CLI command
    ‘args’: [‘eks’, ‘get-token’, ‘–cluster-name’, cluster_name]
    }
    }
    }]

    Login or Signup to reply.
  2. 0

    The 403 Forbidden error in your Lambda function when trying to create a Kubernetes job on your EKS cluster typically stems from an authorization issue. Since you’ve updated the aws-auth ConfigMap and granted system:masters permissions, there are a few more areas to check:

    1. IAM Role Permissions:

    Ensure that the IAM role associated with your Lambda function has the correct policies attached. The role should allow the Lambda function to interact with EKS and perform actions on Kubernetes resources. Verify the following:

    • EKS Permissions: The IAM role should have the necessary permissions to access EKS and fetch the cluster details. Example policy:

      {
    "Effect": "Allow",
    "Action": [
        "eks:DescribeCluster",
        "eks:ListClusters",
        "eks:ListNodegroups",
        "eks:ListFargateProfiles"
    ],
    "Resource": "*"
}

    • STS Permissions: For token authentication using aws eks get-token, ensure the IAM role can call sts:AssumeRole.

    2. AWS CLI in Lambda:

    The error [Errno 2] No such file or directory: 'aws' suggests the AWS CLI is not installed or accessible within the Lambda environment. The aws eks get-token command relies on the AWS CLI.

    Solution:

    • Package the AWS CLI with your Lambda function.
    • Alternatively, use the boto3 EKS client to fetch the token programmatically.

    Example using boto3 for token retrieval:

    from botocore.signers import RequestSigner

def get_eks_token(cluster_name):
    eks_client = boto3.client('eks')
    cluster_info = eks_client.describe_cluster(name=cluster_name)
    cluster_id = cluster_info['cluster']['name']
    role_arn = cluster_info['cluster']['identity']['oidc']['issuer'].split('/')[4]
    
    signer = RequestSigner(
        service_id='sts',
        region_name=boto3.session.Session().region_name,
        signing_name='sts',
        signature_version='v4',
        credentials=boto3.Session().get_credentials(),
        event_hooks=None
    )
    
    params = {
        'method': 'GET',
        'url': f'https://sts.amazonaws.com/?Action=GetCallerIdentity&Version=2011-06-15',
        'body': {},
        'headers': {'x-k8s-aws-id': cluster_id},
        'context': {}
    }
    
    signed_url = signer.generate_presigned_url(
        params,
        region_name=boto3.session.Session().region_name,
        expires_in=60,
        operation_name='get-caller-identity'
    )
    
    token = f"k8s-aws-v1.{base64.urlsafe_b64encode(signed_url.encode()).decode()}"
    return token

    This token can be used in the Kubernetes configuration.

    3. Kubernetes RBAC Permissions:

    Ensure that the Kubernetes Role or ClusterRole bound to the IAM role has the necessary permissions to create jobs in the f-design namespace.

    • Role or ClusterRole Binding Example: 
      kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: lambda-role-binding
  namespace: f-design
subjects:
- kind: User
  name: arn:aws:iam::111122223333:role/lambdarole
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: job-creator
  apiGroup: rbac.authorization.k8s.io

    4. Testing and Validation:

    • Use kubectl with the same IAM role to ensure it can create jobs manually.
    • Check logs and AWS CloudTrail for any unauthorized or failed requests.

    5. Debugging Tips:

    • Ensure the Lambda execution role is correctly assumed and has the necessary trust relationship.
    • Double-check the aws-auth ConfigMap for formatting issues or syntax errors. The role and user mappings should be correct and applied properly.

    Check these areas systematically to ensure the Lambda function can authenticate and perform the required Kubernetes operations. Properly configuring IAM, AWS CLI access, and Kubernetes RBAC should resolve the 403 Forbidden error.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search