I have a python notebook running the following imports on a DataBricks cluster %pip install presidio_analyzer %pip install presidio_anonymizer import spacy.cli spacy.cli.download("en_core_web_lg") nlp = spacy.load("en_core_web_lg") import csv import pprint import collections from typing import List, Iterable, Optional, Union, Dict import…
I have a DataFrame with a single column which is a struct type and contains an array. users_tp_df.printSchema() root |-- x: struct (nullable = true) | |-- ActiveDirectoryName: string (nullable = true) | |-- AvailableFrom: string (nullable = true) |…
We are facing an issue with reading and writing streaming data into the target location. we are working with some JSON telemetry data for tracking steps. New data files land in our delta lake every 5 seconds. Need a way…
I am working on something where I have a SQL code in place already. Now we are migrating to Azure. So I created an Azure databricks for the piece of transformation and used the same SQL code with some minor…
I've seen many iterations of this question but cannot seem to understand/fix this behavior. I am on Azure Databricks working on DBR 10.4 LTS Spark 3.2.1 Scala 2.12 trying to write a single csv file to blob storage so that…
I have the .jpg pictures in the data lake in my blob storage. I am trying to load the pictures and display them for testing purposes but it seems like they can't be loaded properly. I tried a few solutions…
I'm trying to set up the external location for the unity catalog. it was able to connect to storage while trying to test a connection in which storage access is limited to selected vnets and ips. but I'm getting a…
I have a Databricks workspace through Azure, and I use the Databricks Job API to run some python scripts that create files. I want to know if I can retrieve these files created after the job is finished running. In…
I have 2 terraform scripts. One of them creates an Azure VM with a network and 1 subnet. The second TF script creates databricks and 2 additional subnets in the Azure VM network to connect both Databricks and VM. The…
I am trying to ingest 2 csv files into a single spark dataframe. However, the schema of these 2 datasets is very different, and when I perform the below operation, I get back only the schema of the second csv,…