Flask application --> Capturing the value entered via web form and passing it as a parameter to trigger a databricks job. Tried passing the parameters as follows : 1- payload = { > 'notebook_params': { > 'customer_name': customer_name > }…
Question: When doing a Purview scan on Azure Databricks, I get the following error. What could be the cause of the error and how we can fix it? Remarks: For safety, I have replaced the 16 digits in the error…
I'm using Azure Databricks notebook to read a excel file from a folder inside a mounted Azure blob storage. The mounted excel location is like : "/mnt/2023-project/dashboard/ext/Marks.xlsx".2023-project is the mount point and dashboard is the name of the container. When…
i was able to get row values from delta table using foreachWriter in spark-shell and cmd but while writing the same code in azure databricks it doesn't work. val process_deltatable=read_deltatable.writeStream.foreach(new ForeachWriter[Row] { def process(value: Row):Unit ={ val Telemetery= value.getString(0) println(Telemetery)…
I have an Excel file in the azure datalake ,I have read the excel file like the following ddff=spark.read.format("com.crealytics.spark.excel") .option("header", "true") .option("sheetName","__all__") .option("inferSchema","true") .load("abfss://[email protected]/file.xlsx") Now Iam confused how to get just the sheetnames from that Excel file,is there any direct…
I am using azure databricks notebook(python). I am trying to access the azure blob storage container files to azure databricks. To access the file I am trying to use the key Vault as use of hard coded account name and…
I am trying to access Azure Data Lake Storage Gen2 with a Service Principal via Unity Catalog. Managed Identity is added with Contributor Role assigned to the storage account Managed Identity is added as a Storage Credential the storage container…
What will be possible solution for this problem? Using Pi-spark language in notebooks, Does any script be using for this problem?
I need to Maintain Folder Structure to store files in yyyy/MM/DD format and I am getting date like this "2021-12-01T00:00:00Z" I need to Extract year from the date and to store in one variable and need to extract Month from…
I am using the below code to get some information in the Azure Databricks notebook, but runOutput isn't appearing even after the successful completion of the notebook activity. Code that I used. import json dbutils.notebook.exit(json.dumps({ "num_records" : dest_count, "source_table_name" :…