I want to mount my ADLS2 storage into a Databricks workspace without sharing the data with every user in the workspace. We have multiple teams working on different clusters in the same workspace so we want to keep the data…
I have a Spark Dataframe which contains in every single row two items: a file name (with an extension, like for instance .jpg), and the content of the file in bytes. I would like to write a process that takes…
I'm trying to identify/get the most recently added files and accordingly, transfer the information to an Azure SQL Database table. The problem is that the new files are added to subfolders and I don't know how to search within them.…
I have configured event trigger for blob storage folder like container/daily_files . Let us say file1.txt and file2.txt will land simultaneously so it will trigger two instances of the same pipeline. I want to control this and execute with only…
I got a SAS token which was created for a specific folder on my Azure Datalake Gen2. The goal is, to download the folder with all its contents. I understand, that I can create a DataLakeServiceClient, a FileSystemClient, or a…
I'm not sure why, but calling this DataLakeDirectoryClient.CreateSubDirectoryAsync function creates a sub directory and a zero byte file of the same name. I only want it to create the sub directory. Note: I can see the zero byte file in…
I am trying to upgrade my Azure Storage Account to the Gen2 Data Lake. I am running the 3 step process in the UI, but when I get to step 2 (validation), it just sits at 0% and never progresses.…
com.databricks.common.client.DatabricksServiceHttpClientException: PERMISSION_DENIED: Invalid permissions on the specified KeyVault https://azkv*.vault.azure.net/. Wrapped Message: Status code 403, {"error":{"code":"Forbidden","message":"Caller is not authorized to perform action on resource.rnIf role assignments, deny assignments or role definitions were changed recently, please observe propagation time.rnCaller: name=AzureDatabricks;appid=2vf8v4a6-3304-4ab8-85cb-cd0e6f879c1d;oid=4e924d0e-ad49-4acc-baec-6d612e920502;iss=https://sts.windows.net/552af62d-5878-4131-8695-1e87b0f89945/rnAction: 'Microsoft.KeyVault/vaults/secrets/getSecret/action'rnResource: '/subscriptions/6b60a61d-3e92-4501-8794-fd7725066113/resourcegroups/streamproject/providers/microsoft.keyvault/vaults/azkv*/secrets/clientsecret'rnAssignment:…
I have a pipeline to delete some folders from ADLS. My folder structure is as below. /raw/MainFolder/SubfolderA/20230430/File.parquet /raw/MainFolder/SubfolderA/20230415/File.parquet /raw/MainFolder/SubfolderA/20230410/File.parquet /raw/MainFolder/SubfolderB/20230430/File.parquet /raw/MainFolder/SubfolderB/20230420/File.parquet /raw/MainFolder/SubfolderB/20230405/File.parquet The pipeline is currently deleting the File.parquet under the date folder. But I need it to delete the…
I am trying to delete folders that are older than 7 days from ADLS. The folders I want to delete have date as name and are under subfolders of the main folder. Please see below example for folder structure (raw…