Question 322321 in Azure Question posted in
The official documentation to get you started can be found here.

LightIngest to Azure Data Explorer

I am using lightIngest to ingest data from my blob in Azure to Azure Data Explorer. I am having the Storage Blob Data Reader role assignment under my Azure Data Explorer Cluster. I am able to read and list the files, however I couldn’t download the blob to ingest it into Azure Data Explorer Database.

This is the error message: Failure details: Failed to download blob: Access to persistent storage path

My question is:

  1. How do lightingest actually ingesting the data? I couldnt find it from the documentation
  2. What roles do I need to overcome the above error? Do I need the Storage Blob Data Contributor role or there is something wrong on my command

Side note: I have database admin permisson on the database in azure data explorer

My command:
LightIngest "https://{XXX}.kusto.windows.net;Fed=True" -database:XXX -table:XXX -source:"https://{storage account}.blob.core.windows.net/{container}/{blob path}/{blob path}/{blob path}/{blob path};managed_identity=system" -pattern:"*.parquet" -format:"parquet" -ingestWithManagedIdentity:"system" -connectToStorageWithUserAuth:"PROMPT"

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    I found an answer to this. Apparently you will need an private managed end point, instead of a managed end point


  2. 0

    Answering your questions

    1. LightIngest relies on Kusto Managed Tables to ingest data efficiently. When you use LightIngest, it generates Kusto commands to ingest the data into Managed Tables.
    2. To troubleshoot the error, chek that you have the permission I mentioned below
      Storage Blob Data Reader
      Storage Blob Data Contributor
      Database Admin Permission
    3. I guess its good to use Managed Identity (-ingestWithManagedIdentity:"system").

    Try this code I have modified it a bit

    LightIngest "https://{cluster}.kusto.windows.net;Fed=True" `
  -database:YourDatabase `
  -table:YourTable `
  -source:"https://{storage account}.blob.core.windows.net/{container}/{blob path}/*.parquet;managed_identity=system" `
  -format:"parquet" `
  -ingestWithManagedIdentity:"system" `
  -connectToStorageWithUserAuth:"PROMPT"

    If you still face the error I recommend going through the Microsoft doc, ill give u the link

    https://learn.microsoft.com/en-us/search/?terms=LightIngest

    https://learn.microsoft.com/en-us/search/?terms=Data%20Explorer%20Storage%20Blob%20Data%20Ingestion

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search