I'm using Azure Databricks notebook to read a excel file from a folder inside a mounted Azure blob storage. The mounted excel location is like : "/mnt/2023-project/dashboard/ext/Marks.xlsx".2023-project is the mount point and dashboard is the name of the container. When…
I have a requirement where a filed value is "CAH132242F0A" from this i have to come up with a date. so here "13" is the year and so yyyy = "2013" and "224" th day. so i have to extract…
The Subscription itself, which includes Azure Databricks created in Directory A of account A, has been transferred to directory B of account B. A Compute cluster and Workflow were already created in directory A of account A. It for running…
I have my my dbt-project configured and running for dbt-postgres. Is there an easy way to port it to dbt- databricks without having to rewrite the models. Context: I am building a dbt demo on postgres, databricks and Azure SQL…
I'm trying to decrypt a string in databricks using aes_decrypt function with GCM mode. But, it doesn't seem to work with IV. Didn't find any details about how to use IV with aes_decrypt function : https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/functions/aes_decrypt?source=recommendations Here are the details…
I have an Excel file in the azure datalake ,I have read the excel file like the following ddff=spark.read.format("com.crealytics.spark.excel") .option("header", "true") .option("sheetName","__all__") .option("inferSchema","true") .load("abfss://[email protected]/file.xlsx") Now Iam confused how to get just the sheetnames from that Excel file,is there any direct…
I have the following scala code in Azure Databricks that creates a dataframe with the list of files and file properties for an specific folder. Can someone help me to transform that code to recursively navigate to each sub folder…
I have a python notebook running the following imports on a DataBricks cluster %pip install presidio_analyzer %pip install presidio_anonymizer import spacy.cli spacy.cli.download("en_core_web_lg") nlp = spacy.load("en_core_web_lg") import csv import pprint import collections from typing import List, Iterable, Optional, Union, Dict import…
I have a DataFrame with a single column which is a struct type and contains an array. users_tp_df.printSchema() root |-- x: struct (nullable = true) | |-- ActiveDirectoryName: string (nullable = true) | |-- AvailableFrom: string (nullable = true) |…
We are facing an issue with reading and writing streaming data into the target location. we are working with some JSON telemetry data for tracking steps. New data files land in our delta lake every 5 seconds. Need a way…