Unity Catalog is the Azure Databricks data governance solution for the Lakehouse. Whereas, Microsoft Purview provides a unified data governance solution to help manage and govern your on-premises, multicloud, and software as a service (SaaS) data. Question: In our same…
In Azure, I have a virtual network (vNET) with following settings: Address space: 10.200.0.0/20 subnet: 10.200.0.0/24 Inside the above vNET, I am trying to deploy an Azure Databricks with the following Network settings: Public Subnet CIDR Range: 10.200.15.0/20 Private Subnet…
I'm starting a Databricks notebook from ADF to do some preprocessing tasks. The cluster of this notebook is usually not running and should only run, when the ADF pipeline is running as well. But it takes several minutes for the…
I am working on something where I have a SQL code in place already. Now we are migrating to Azure. So I created an Azure databricks for the piece of transformation and used the same SQL code with some minor…
We are porting some code from SSIS to Python. As part of this project, I'm recreating some packages but I'm having issues with the database access. I've managed to query the DB like this: employees_table = (spark.read .format("jdbc") .option("url", "jdbc:sqlserver://dev.database.windows.net:1433;database=Employees;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;")…
I have Azure Databricks workspace hidden inside VNET and configured with Private Link, following instruction from Microsoft. Now I'm trying to do SCIM provisioning with this workspace and I'm getting following error. When I tried with workspace without Private Link…
Description : I have a list of column names which I need. I want to check if all these columns names are present in dataframe.if some columns are present then use those columns and make a generic code like Df1=df.select(df[column1],df(column2])…
We have our delta-lake from which we can query data. We also connected it with Power BI to make interactive dashboards. It's in production. Now we want to use Azure Purview to avail all the data governance and data catalog…
I'm working on some data (~200GB) using spark in azure databricks. I am able to read the dataset (from blob storage) and modify it in various ways. However, every time I try to store it, either through .saveAsTable() or .csv()…
I have to ingest data from ADLS gen2 to tables in ADX. I first tried to use ADF, but considering the run time and everything, this was an really inefficient way to do it. I think it would be so…