We have our delta-lake from which we can query data. We also connected it with Power BI to make interactive dashboards. It's in production. Now we want to use Azure Purview to avail all the data governance and data catalog…
I'm working on some data (~200GB) using spark in azure databricks. I am able to read the dataset (from blob storage) and modify it in various ways. However, every time I try to store it, either through .saveAsTable() or .csv()…
I have to ingest data from ADLS gen2 to tables in ADX. I first tried to use ADF, but considering the run time and everything, this was an really inefficient way to do it. I think it would be so…
In Azure Databricks, I am able to add different git repos in repos filed manually through console. But not able to find any way to automatically sync new files if I committed anything in source repos. Is there any service…
I am trying to connect to an external private MySQL Database from Azure Databricks. I have tried with sqlalchemy and pymysql: pymysql: conn = pymysql.connect( host='.....', user='.....', password = "....", db='....') cur…
please can some one guide me how to Establish Connection to SQL server using manage identity in data bricks using manage identity object id I am using following code ''' from azure.core.exceptions import ClientAuthenticationError from azure.identity import DefaultAzureCredential # Create…
I m trying to read csv file in databricks using pyspark where it has columns shuffled instead of A ,B, C it will randomly arranged like C,A,B i tried using map() , it throws error 'cannot pickle '_thread.RLock' object' i…
Save VM cost when using Rest API deploying models for online inference ADB allows us to deploy the models for online inference through a REST API. To that aim ADB creates a VM dedicated to serve a specific model. Data…
I have installed the following jar in databricks "com.redislabs:spark-redis_2.12:2.5.0". And trying create a spark session with the respective authentications Below is the code where I create a spark session with creds redis= SparkSession.builder.appName("redis_connection").config("spark.redis.host", "hostname").config("spark.redis.port", "port").config("spark.redis.auth", "pass").getOrCreate() But when I try…