azure-data-lake

How can I convert string output of Set Variable activity to .csv or .txt file in Azure Data Factory

I want to take the output of the Set Variable activity which is in String format and convert it into a .csv or .txt format. [Set Variable output: { "name": "Data_String", "value": "[{"Name":"Balaji","EnabledFlag":"Y","CategoryId":"300000290956003","LastUpdateDate":"19-09-2023","CreationDate":"19-09-2023","Entity":"027"},{"Name":"Ahmed","EnabledFlag":"Y","CategoryId":"300000290956073","LastUpdateDate":"19-09-2023","CreationDate":"19-09-2023","Entity":"027"}, } ] Here the output of the…

VIEW QUESTION

Does Azure-ADF Delete Activity delete Folders?

I have a pipeline to delete some folders from ADLS. My folder structure is as below. /raw/MainFolder/SubfolderA/20230430/File.parquet /raw/MainFolder/SubfolderA/20230415/File.parquet /raw/MainFolder/SubfolderA/20230410/File.parquet /raw/MainFolder/SubfolderB/20230430/File.parquet /raw/MainFolder/SubfolderB/20230420/File.parquet /raw/MainFolder/SubfolderB/20230405/File.parquet The pipeline is currently deleting the File.parquet under the date folder. But I need it to delete the…

VIEW QUESTION

Azure – Auto create table in ADF copy activity for ODBC Sink Dataset (Not Working)

I am trying to perform copy activity between ADLS dataset and ODBC sync dataset (POSTGRESQL) But, table is not creating automatically in target dataset and error coming like Failure happened on 'Sink' side. ErrorCode=UserErrorOdbcOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ERROR [42P01] ERROR: relation "semantic_dev.dim_storage_location" does not…

VIEW QUESTION

Pyarrow slice pushdown for Azure data lake

I want to access Parquet files on an Azure data lake, and only retrieve some rows. Here is a reproducible example, using a public dataset: import pyarrow.dataset as ds from adlfs import AzureBlobFileSystem abfs_public = AzureBlobFileSystem( account_name="azureopendatastorage") dataset_public = ds.dataset('az://nyctlc/yellow/puYear=2010/puMonth=1/part-00000-tid-8898858832658823408-a1de80bd-eed3-4d11-b9d4-fa74bfbd47bc-426339-18.c000.snappy.parquet',…

VIEW QUESTION

Azure – Timeout error while uploading to large file in adls

I need to upload a 200 mb file to adls using python. I'm using the code provided in the official documentation - https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-directory-file-acl-python?tabs=azure-ad While calling the following function for upload - def upload_file_to_directory_bulk(): try: file_system_client = service_client.get_file_system_client(file_system="system") directory_client = file_system_client.get_directory_client("my-directory")…

VIEW QUESTION
Back To Top
Search