I have to ingest data from ADLS gen2 to tables in ADX.
I first tried to use ADF, but considering the run time and everything, this was an really inefficient way to do it.
I think it would be so much easier if there is a way to ingest data into ADX via databricks.
I only can access ADLS through Service Pricipals so I can’t directly connect to the storage account from ADX.
Any help will be much appreciated!
2
Answers
If it is for continuous ingestion consider creating an Event Grid data connection
If it is for one-time ingestion consider using the "lightIngest" tool, and provide the applicable connection string for your use case.
I would also recommend the Event Grid Data connection. Please note that you can set up this data connection using managed identities.