I am using azure databricks notebook(python). I am trying to access the azure blob storage container files to azure databricks. To access the file I am trying to use the key Vault as use of hard coded account name and access is not good practice but I am getting following error:
(Forbidden) The user, group or application 'appid=-" ;iss=https://sts.windows.net/****/' does not have secrets get permission on key vault '***Key;location=eastus'. For help resolving this issue, please see https://go.microsoft.com/fwlink/?linkid=2125287
Code: Forbidden
Message: The user, group or application 'appid=******;oid=*****;iss=https://sts.windows.net/****/' does not have secrets get permission on key vault '****Key;location=eastus'. For help resolving this issue, please see https://go.microsoft.com/fwlink/?linkid=2125287
Inner error: {
"code": "AccessDenied"
}
I have added secretscope to databricks. here is the screenshot of access policies:
here is my full code:
from azure.identity import DefaultAzureCredential
from azure.keyvault.secrets import SecretClient
from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient
import pandas as pd
import io
import pyspark.sql
from datetime import datetime, timedelta
#set Azure Key Vaults details
key_vault_name = '***key'
secret_name = '******'
credential = DefaultAzureCredential()
#create Secret Client
secret_client = SecretClient(vault_url=f"https://{key_vault_name}.vault.azure.net/", credential=credential)
#define Azure Blob Storage connection details
storage_account_name = secret_client.get_secret('storage-account-name').value
storage_account_access_key = secret_client.get_secret('storage-account-access-key').value
container_name = '*****'
refined_container_name = '*****'
connection_string = secret_client.get_secret(secret_name).value
print('storage_account_name = ',storage_account_name)
print('storage_account_access_key = ',storage_account_access_key)
print('connection_string = ', connection_string)
3
Answers
On the Key Vault, add an access policy for the application id and add the "Get" secret permission.
You need to use dbutils(1)(2) to interact with databricks secret scoped instead of Azure key vault SDK.
(1) https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-utils#dbutils-secrets
(2) https://learn.microsoft.com/en-us/azure/databricks/security/secrets/example-secret-workflow#secret-example-notebook
I reproduce same thing in my environment .I got same error.
To resolve above error .Please follow below steps :
Step 1: Go to azure key vault -> Access policies -> + create and provide enough permissions.
Step2 : Create Secret Scope -> Go to Azure_Vault -> Enter the Vault_URI .(For Example, Vault_URI:
https://<key_vault_name.>.vault.azure.net/
) and Resource ID -> Go to Properties tab of an Azure Key Vault in your Azure portal you get both Vault URI and Resource ID.Code:
Output: