I am trying to write data to Cassandra table (cosmos DB) via Azure DBR job (spark streaming). Getting below exception: Query [id = , runId = ] terminated with exception: Failed to open native connection to Cassandra at {<name>.cassandra.cosmosdb.azure.com:10350} ::…
So how we can fetch the secrets or keys for Azure Data Factory , logic apps, Azure Synapse and Azure DataBricks if we disable the public access for keyvault. I found a solution for AppServices and FunctionApps by using outbound…
I have a DataFrame with a single column which is a struct type and contains an array. users_tp_df.printSchema() root |-- x: struct (nullable = true) | |-- ActiveDirectoryName: string (nullable = true) | |-- AvailableFrom: string (nullable = true) |…
Unity Catalog is the Azure Databricks data governance solution for the Lakehouse. Whereas, Microsoft Purview provides a unified data governance solution to help manage and govern your on-premises, multicloud, and software as a service (SaaS) data. Question: In our same…
In Azure, I have a virtual network (vNET) with following settings: Address space: 10.200.0.0/20 subnet: 10.200.0.0/24 Inside the above vNET, I am trying to deploy an Azure Databricks with the following Network settings: Public Subnet CIDR Range: 10.200.15.0/20 Private Subnet…
I'm starting a Databricks notebook from ADF to do some preprocessing tasks. The cluster of this notebook is usually not running and should only run, when the ADF pipeline is running as well. But it takes several minutes for the…
I am working on something where I have a SQL code in place already. Now we are migrating to Azure. So I created an Azure databricks for the piece of transformation and used the same SQL code with some minor…
We are porting some code from SSIS to Python. As part of this project, I'm recreating some packages but I'm having issues with the database access. I've managed to query the DB like this: employees_table = (spark.read .format("jdbc") .option("url", "jdbc:sqlserver://dev.database.windows.net:1433;database=Employees;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;")…
I have Azure Databricks workspace hidden inside VNET and configured with Private Link, following instruction from Microsoft. Now I'm trying to do SCIM provisioning with this workspace and I'm getting following error. When I tried with workspace without Private Link…
Description : I have a list of column names which I need. I want to check if all these columns names are present in dataframe.if some columns are present then use those columns and make a generic code like Df1=df.select(df[column1],df(column2])…