I am new to Databricks and would appreciate any help :) I am using Databricks for an ELT project to read from Blob storage transform the data to Dataframes, and write the data to another blob storage(in essence a Lighthouse…
I have my project file siting in the path /Workspace/folder1/subfolder1/myproject This is how I have structured my project files. Structure of Module folder: enter image description here baseMLPipeline is my entry file where I am importing the file in the…
I am new to ADF, I am trying to connect with SQL Server through SSMS setup from my local system. For this, I have created a keyvault in Azure and stored username and passsword into that. Also I have setup…
Hi I am new and trying to learn Airflow on AWS, I am creating the Environment and got the error that Maximum worker count must be between 1 and 0. Could someone please help me to fix it. Thank you…
Using autoloader, I am reading some continues data from storage to Databricks Delta Live table. The declaration of data pipeline is as follows. import dlt from pyspark.sql.functions import * from pyspark.sql.types import * sch = "StructType([StructField('Date', StringType(), True), StructField('machine', StringType(),…
I have an array output (len = 503 ) from the lookup value activity which is basically fetching strings from a CSV file. These strings are parameters for my dataflow activity within the for each. Current adf pipeline: I now…