skip to Main Content

I am trying to connect Azure SQL with AKS using managed Identity but this doesnt seems to work.
I have been checking numerous blogs and I am as confused as I started.
I am using aad-pod-identity:

https://github.com/Azure/aad-pod-identity

https://blog.seblab.be/posts/configure-aad-pod-ientity-on-aks/
https://medium.com/devopsturkiye/azure-kubernetes-service-aks-mimarisinde-aad-pod-identity-943effa54a39
https://medium.com/devopsturkiye/azure-kubernetes-service-aks-mimarisinde-aad-pod-identity-943effa54a39

I tried to create an admin user as well, using this query:

/* Create db contained user for user assigned identity
   you can use the principal name or display name */
CREATE USER sqlpodid FROM EXTERNAL PROVIDER;

/* Assign the needed security roles to this db contained user */
ALTER ROLE db_datareader ADD MEMBER sqlpodid;
ALTER ROLE db_datawriter ADD MEMBER sqlpodid;

/* Check that the assignation is ok */
SELECT DP1.name AS DatabaseRoleName,   
    isnull (DP2.name, 'No members') AS DatabaseUserName   
FROM sys.database_role_members AS DRM  
RIGHT OUTER JOIN sys.database_principals AS DP1  
    ON DRM.role_principal_id = DP1.principal_id  
LEFT OUTER JOIN sys.database_principals AS DP2  
    ON DRM.member_principal_id = DP2.principal_id  
WHERE DP1.type = 'R'
ORDER BY DP1.name;
apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentityBinding
metadata:
 name: sqlaadbinding1
spec:
 AzureIdentity: my-identity
 Selector: connectsqlserver
kind: AzureIdentity
metadata:
 name: my-identity
spec:
 type: 0
 ResourceID: /subscriptions/<sub id>/resourcegroups/<MC_resource_group/providers/Microsoft.ManagedIdentity/userassignedIdentities/sqlpodid
 ClientID: <cliendId of sqlpodid>

The MIC pod repeatedly showing the below log:

reconciling identity assignment for [/subscriptions/<sub id>/resourcegroups/<MC_resource_group>/providers/Microsoft.ManagedIdentity/userassignedIdentities/sqlpodid] on node aks-agentpool-vmss

The NMI pod is showing:

server.go:427] failed to get matching identities for pod: default/pod, error: clientID in request: id##### REDACTED #####id, getting assigned identities for pod default/pod in CREATED state failed after 16 attempts, retry duration [5]s, error: <nil>. Check MIC pod logs for identity assignment errors
2023-07-27T17:15:05.971574924Z I0727 17:15:05.971454       1 server.go:239] status (404) took 80016338075 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=10.244.0.32

I dont know what is the redacted clientId, I tried to search in agentpool managed identity, user managed identity, AKS cluster or ressource group. I didnt find it anywhere.

Other than that I tried to give my managed Identity a role assignment as well.
az role assignment create --role "Managed Identity Operator" --assignee <CLIENT ID OF AKS CLUSTER> --scope <FULL RESOURCE ID OF MANAGED IDENTITY>

for me in AKS cluster the serviePrinipalProfile.clientId = msi.
so I chose the clientid from: identityProfile.kubeletidentity.clientId.

The pod definition is simple. Its a simple Spring boot App which has
Spring.datasource.url=url: "jdbc:sqlserver://<SRV>.database.windows.net:1433;database=<DB>;authentication=ActiveDirectoryMSI;msiClientId=<MI ID>;selectMethod=cursor;responseBuffering=full;sendStringParametersAsUnicode=false;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;"

where MSI clientId is the Managed Identity "sqlpodid"’s clientid. Some places its mentioned as PrincipalId or ObjectId I am not sure what goes here.

This is my pod yaml:

apiVersion: v1
kind: Pod
metadata:
  name: pod
  labels:
    aadpodidbinding: connectsqlserver
spec:
  containers:
...

nothing seems to work.

2

Answers


  1. Comparing the official documentation about Azure AD Pod Identity for AKS versus the role assignment you mentioned you did, the following role assignment to the cluster system assigned managed identity may be missed: "Virtual Machine Contributor". Without the proper role assignments, your Azure cluster will not have the correct permission to assign and un-assign identities from the underlying virtual machines (VM) or virtual machine scale sets (VMSS).

    Please make sure the user assigned managed identity that accesses the Azure SQL database is created within the same resource group where the AKS cluster was created.

    In addition, please consider adding AzureServicesAuthConnectionString to the ConfigMap.

    Here you may find another useful article about this topic.

    Login or Signup to reply.
    1. Enable workload identities in the AKS cluster (https://learn.microsoft.com/en-us/azure/aks/workload-identity-deploy-cluster).

      az aks update -g "${RESOURCE_GROUP}" -n myAKSCluster –enable-oidc-issuer –enable-workload-identity

    2. Create a managed identity

      az identity create –name "${USER_ASSIGNED_IDENTITY_NAME}" –resource-group "${RESOURCE_GROUP}" –location "${LOCATION}" –subscription "${SUBSCRIPTION}"

    3. Retrieve the oidc issuer url:

      $oidcIssuer="$(az aks show -n $clusterName -g $resourceGroupName –query "oidcIssuerProfile.issuerUrl" -otsv)"

    4. Replace the double underscore variable in the following YAML template with the proper values. Note: serviceAccountName is up to you and make sure it’s in the same namespace as the service that will use it:

      apiVersion: v1
      kind: ServiceAccount
      metadata:
      annotations:
      azure.workload.identity/client-id: clientID
      azure.workload.identity/tenant-id: tenantId
      labels:
      azure.workload.identity/use: "true"
      name: serviceAccountName
      namespace: namespace

    5. Create a federated credential. This links your service account (YAML above) with your managed identity:

      az identity federated-credential create –name "$serviceName-$namespace" –identity-name $identityName –resource-group $rg –issuer $oidcIssuer –subject "system:serviceaccount:${namespace}:${serviceAccountName}"

    6. Make sure you have the helm charts for azure-workload-identity:

      helm repo add azure-workload-identity https://azure.github.io/azure-workload-identity/charts
      helm repo update
      helm install workload-identity-webhook azure-workload-identity/workload-identity-webhook –namespace azure-workload-identity-system –create-namespace –set azureTenantID="tenantId"

    7. Navigate to your managed identity, you should see your federated credential. Next, you need to make sure your YAML has the following parameters:

      azure.workload.identity/use: "true"
      serviceAccountName: serviceAccountName

    Above is how you associate your service to the federated credentials using workload identities.

    Lastly, getting the token for making requests to Azure SQL behave similarly to retrieving default credentials with the code. Same as your identity authenticating to Azure SQL using system creds (tokens created on your machine when logging in with your account). You added the user properly to the database.

    Culminates into the following script…

    #This script creates and assigns workload identities in the AKS cluster. Workload identities need to be enabled, and also the managed identity needs
    #to be created. The managed identity created by Teraform will then be "federated" for workload identity assignment to pods.
    param(
        [Parameter(Mandatory=$true)]
        [string]
        $clusterName,
        [Parameter(Mandatory=$true)]
        [string]
        $rg,
        [Parameter(Mandatory=$true)]
        [string]
        $serviceName,
        [Parameter(Mandatory=$true)]
        [string]
        $namespace,
        [string]
        $prefix="test-",
        [string]
        $identityName,
        [string]
        $tenantId
    )
    class CustomException : Exception {
        [string] $additionalData
    
        CustomException($Message, $additionalData) : base($Message) {
            $this.additionalData = $additionalData
        }
    }
    
    $clientID = "$(az identity show --name $identityName --resource-group $rg --query 'clientId' -otsv)"
    
    $serviceAccountName = "$prefix$serviceName-sa"
    $yaml = `apiVersion: v1
    kind: ServiceAccount
    metadata:
      annotations:
    azure.workload.identity/client-id: __clientID__ 
    azure.workload.identity/tenant-id: __tenantId__
      labels:
    azure.workload.identity/use: "true"
      name: __serviceAccountName__ 
      namespace: __namespace__ `
    
    #overwrite tokens in the template and write out to a new file
    $yaml = $yaml.Replace("__clientID__", $clientID);
    $yaml = $yaml.Replace("__serviceAccountName__", $serviceAccountName);
    $yaml = $yaml.Replace("__namespace__", $namespace);
    $yaml = $yaml.Replace("__tenantId__", $tenantId);
    Set-Content -Path .service-account.yaml -Value $yaml
    
    #create the service account that needs to be bound to the federated identity
    kubectl apply -f .service-account.yaml -n $namespace
    
    #Clear content
    Set-Content -Path .service-account.yaml -Value ""
    
    try {
        $oidcIssuer="$(az aks show -n $clusterName -g $rg --query "oidcIssuerProfile.issuerUrl" -otsv)"
        if($null -eq $oidcIssuer -or "" -eq $oidcIssuer)
        {
            #oidc does not exist in the cluster. It needs to be enabled. Check if cluster addon is enabled. First check az version
            $azVersions = az version | ConvertFrom-Json
            $cliVersion = $azVersions.'azure-cli';
            #cliVersion needs to be >= 2.40.0
    
            if(([Version] $cliVersion) -lt ([Version] "2.40.0")){
                Write-Host "CLI needs upgraded based on the dependencies here... https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview";
                #az upgrade --yes --all --quiet
                throw [CustomException]::new("Azure CLI $cliVersion needs to be upgraded to >=2.40.0 to leverage Workload Identities and Federated credentials with the azure cli.","") > $PSScriptRooterror.log
            }
            
            #see if the WorkloadIdentityPreview feature is enabled and registered
            $featureState = az feature show --namespace "Microsoft.ContainerService" --name "EnableWorkloadIdentityPreview" | ConvertFrom-Json
            if($featureState.properties.state -ne "Registered"){
                #try to enable and register the 'EnableWorkloadIdentityPreview' feature flag
                Write-Host "Feature flag EnableWorkloadIdentityPreview is not registered... attempting to register..."
                az feature register --namespace "Microsoft.ContainerService" --name "EnableWorkloadIdentityPreview";
                #We have to wait a minute or two for the features to register before we can continue.
                Start-Sleep -Seconds 600;
                az provider register --namespace Microsoft.ContainerService
                #Verify feature is in a registered state
                $featureState = az feature show --namespace "Microsoft.ContainerService" --name "EnableWorkloadIdentityPreview" | ConvertFrom-Json
                if($featureState.properties.state -ne "Registered"){
                    throw [CustomException]::new("Feature EnableWorkloadIdentityPreview did not finish registering after 120 seconds, please wait and try again. If this fails a second time the verify that Microsoft.ContainerService is enabled in the subscription under 'Features'.","") > $PSScriptRooterror.log
                }
                else{
                    Write-Host "Feature was successfully registered... continuing..."
                }
            }
            #update the cluster to use OIDC and workload identities
            az aks update -g $rg -n $clusterName --enable-oidc-issuer --enable-workload-identity
            #now finally get the OIDC URL
            $oidcIssuer="$(az aks show -n $clusterName -g $rg --query "oidcIssuerProfile.issuerUrl" -otsv)"
            if($null -eq $oidcIssuer -or "" -eq $oidcIssuer){
                throw [CustomException]::new("Still cannot retrieve OIDC URLs. Please wait an hour and try again. Cluster might still be updating.",""); > $PSScriptRooterror.log
            }
        }
    
        az identity federated-credential create --name "$serviceName-$namespace" --identity-name $identityName --resource-group $rg --issuer $oidcIssuer --subject "system:serviceaccount:${namespace}:${serviceAccountName}"
    }
    catch {
        throw [CustomException]::new("Federation setup failed.","") > $PSScriptRooterror.log
    }
    #$oidcIssuer="$(az aks show -n $clusterName -g $rg --query "oidcIssuerProfile.issuerUrl" -otsv)"
    
    try{
        #see if all helm charts exist for workload identity
        $charts = helm repo list -o json | ConvertFrom-Json
        $url = $charts | Where-Object {$_.Name -eq "azure-workload-identity"} | Select-Object -ExpandProperty url
        if($null -eq $url -or "" -eq $url){
            write-host "azure-workload-identity is missing from the helm charts... attempting to add it."
            helm repo add azure-workload-identity https://azure.github.io/azure-workload-identity/charts
            helm repo update
            helm install workload-identity-webhook azure-workload-identity/workload-identity-webhook --namespace azure-workload-identity-system --create-namespace --set azureTenantID="$tenantId"
            Write-Host "Helm charts 'workload-identity-webhook' and 'azure-workload-identity/workload-identity-webhook' successfully added!"
        }
    }
    catch {
        throw [CustomException]::new("Helm setup failed.","") > $PSScriptRooterror.log
    }
    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search