I am trying to write to Snowflake using Snowpark from Azure Databricks and I get the following error. This error does not seem related to Unity Catalog but some configuration. I have made sure I have imported required libraries. df_new.write.mode("append").saveAsTable("MyTableName")…
The following bicep deployment has suddenly started failing and I do not understand why var managedResourceGroupName = 'rg-metadata-${dataBricksName}' resource databrickworkspace 'Microsoft.Databricks/workspaces@2018-04-01' = { name: dataBrickName location: location sku: { name: dataBrickPricingTier } properties: { managedResourceGroupId: managedResourceGroup.id parameters: { enableNoPublicIp: {…
I am facing an issue where the expected date parition folder should be named in format date=yyyymmdd, but instead writing as - Sometimes for each parquet file created in delta path, it's creating a seperate folder. I don't see any…
I am currently developing a Python script to retrieve a comprehensive list of all the jobs that were executed yesterday. However, I'm encountering an issue with the script's pagination mechanism using tokens. Despite my attempts to loop through the pagination…
I have these following inputs. 1.Endpoint 2.Database 3.Collections Based on these inputs needs to establish a connection with Azure Cosmos Db from databricks Tried this approach: installed importted com.azure.cosmos.spark maven package in databricks cluster with the DBR of 11.3LTS and…
I have this tableConfig.json inside a ADLS container based location . It has table specific details. { "tableName":"employee", "databaseName": "dbo", "location" : "/mnt/clean/demo", "colsList" : ["emp_id","emp_name","emp_city"] } Now I want to read that tableConfig.json in azure databricks python notebooks. and…
com.databricks.common.client.DatabricksServiceHttpClientException: PERMISSION_DENIED: Invalid permissions on the specified KeyVault https://azkv*.vault.azure.net/. Wrapped Message: Status code 403, {"error":{"code":"Forbidden","message":"Caller is not authorized to perform action on resource.rnIf role assignments, deny assignments or role definitions were changed recently, please observe propagation time.rnCaller: name=AzureDatabricks;appid=2vf8v4a6-3304-4ab8-85cb-cd0e6f879c1d;oid=4e924d0e-ad49-4acc-baec-6d612e920502;iss=https://sts.windows.net/552af62d-5878-4131-8695-1e87b0f89945/rnAction: 'Microsoft.KeyVault/vaults/secrets/getSecret/action'rnResource: '/subscriptions/6b60a61d-3e92-4501-8794-fd7725066113/resourcegroups/streamproject/providers/microsoft.keyvault/vaults/azkv*/secrets/clientsecret'rnAssignment:…
Flask application --> Capturing the value entered via web form and passing it as a parameter to trigger a databricks job. Tried passing the parameters as follows : 1- payload = { > 'notebook_params': { > 'customer_name': customer_name > }…
Question: When doing a Purview scan on Azure Databricks, I get the following error. What could be the cause of the error and how we can fix it? Remarks: For safety, I have replaced the 16 digits in the error…
I'm using Azure Databricks notebook to read a excel file from a folder inside a mounted Azure blob storage. The mounted excel location is like : "/mnt/2023-project/dashboard/ext/Marks.xlsx".2023-project is the mount point and dashboard is the name of the container. When…