Delete bottom two rows in Azure Data Flow
I would like to delete the bottom two rows of an excel file in ADF, but I don't know how to do it. The flow I am thinking of is this. enter image description here *I intend to filter ->…
Azure – Cant list blobs from Synapse notebook
I am having a bit of a trouble with Synapse notebooks. I want to get a list of blob via pyspark script to dynamically decide which files I want to integrate. I cannot make this thing work in Synapse.. in…
Can PowerBI inherit RLS from Azure Synapse SQL?
I have a Azure Synapse SQL table which contains a list of companies. I have created an access matrix, security predicate and policy to filter data presented based on the user. Create FUNCTION spf.rls_securitypredicate_dimcompany(@Companycode AS nvarchar(50)) RETURNS TABLE WITH SCHEMABINDING…
File path error in pipeline for spark notebook in azure synapse
I have a spark notebook which I am running with the help of pipeline. The notebook is running fine manually but in the pipeline it is giving error for file location. In the code I am loading the file in…
How to set the dependency between two trigger pipelines in azure synapse Analytics
I have a two trigger synapse pipelines one which is scheduled at 03 am cst , What I'm looking now is the Second pipeline should trigger after the completion of the first pipeline i.e after 03 am cst. Is there…
I would like to add the date and time string in addition to the source file name.(Azure Synapse Analytics)
When copying a file from S3 to AzureBlobStorage, I would like to add the date and time string in addition to the source file name. In essence, the S3 folder structure looks like this data/yyyy/mm/dd/files *yyyy=2019-2022, mm=01-12, dd=01-31 And when…
Azure – Does the container in the data lake storage Gen2 account consist of a blob storage, if so why does it differ from the normal blob storage account?
I am new to Azure. I am currently following a tutorial on Azure Synapse Analytics. So far I am learning about how the data is stored. My understanding so far is that the container is contained within a workspace and…
Azure – CI/CD Pipeline is failing for Synapse
I am getting below error when deploying Synapse CI/CD pipeline using Azure DevOps. The deployment was working fine until I added a few more notebooks and it started failing. Notebooks run fine on lower environment. All paths in the notebooks…
One source to many sinks azure synapse pipeline?
I’m using a copy activity in azure synapse pipeline to copy and filter data from containerA/file1.csv to containerB/file2US.csv Similarly I’m using another copy activity to copy and filter data from containerA/file1.csv to containerB/file2IND.csv The same process for different regions. In…
Azure – How to save the file name with current date and time in PySpark?
I have a data frame in PySpark and would like to save the file as a CSV with the current timestamp as a file name. I am executing this in Azure Synapse Notebook and would like to run the notebook…