I have created an AWS MWAA environment and correctly pointed it to an S3 bucket as well as an "dags" folder inside the S3 bucket. I have created a simple dag file (python) and added it to my /dags/ directory…
I have successfully created a mariadb database connection using debezium and kafka When I tried to stream the topic using pyspark this is the output that I get ------------------------------------------- Batch: 0 ------------------------------------------- +------+--------------------------------------------------------------------------------------------------------------------------+ |key |value | +------+--------------------------------------------------------------------------------------------------------------------------+ ||MaxDoe1.4.2.Finalnmysqlmariadbbtruebasecampemployees mysql-bin.000032�r�ȯݭd |…
I am reaching out to you for assistance with some issues I'm encountering in my Databricks environment. I'm hoping the community can provide some guidance to help me resolve these problems. 1. Error with df.display(): When I try to use…
I stumbled upon a good ADF template called Copy new files only by LastModifiedDate. This ADF templates allows to sync two Blob Storage but only one way A --> B. The templates works fairly well. The problem is that when…
I am using Web Activity to send notifications of pipeline run in MS Teams. My requirement is to use minimum number of web activities to reduce the cost. At the same time I want notification in case any of the…
A Python Shell job cannot use more than one DPU. This means that it has a limit of 16 GB of memory. Earlier today, I wired what I considered to be a modest ETL task to AWS Glue with 1…
I have a file which is present as a unstructured txt in ADLS. I want to read the entire file content as text and pass it to a Stored Procedure in Azure data factory. Can this be done through Azure…
I'm working in a simple ETL for our budget on MS Azure Data Factory. This pipeline takes a source from an Excel and has a Sink on MS Azure SQL Database. Already done the integration and it works, but i'm…
Backslash from the json is getting removed after the file is processed by copy activity I have json input file which contains the below object INPUT { "Name": "xyz", "DOB":"14/09/1995" } After i read the files and save using the…
I have this case where we have different queries we want to run in SQL Server then save it in an Excel file. I parameterized the Excel dataset I have then assigned it in the "Sheet name" property but I…