skip to Main Content

I have set up an Azure Synapse Serverless SQL Pool, with the goal of reading data from a Parquet data lake stored in an Azure Data Lake Gen2 container.

Within Synapse Studio, I can create an SQL script as below, which works:

SELECT TOP(50) AVG(Speed)
FROM serverlessdbtest.dbo.CAN2_gnssspeed

I am now trying to do this externally from a Grafana Cloud account. I have done the following:

  1. Added a Microsoft SQL Server data source
  2. In the authentication I used the below settings:

enter image description here

Note here that the user details I have specified come from below:

enter image description here

The above lets me connect to the Azure Synapse database and within a Grafana panel I can e.g. use the builder to show my tables, columns etc. But when I try to execute the aforementioned query in Grafana, I get the below error:

db query error: mssql: External table 'serverlessdbtest.dbo.CAN2_gnssspeed' is not accessible because content of directory cannot be listed.

It seems the issue is that I need to add a Storage Blob Data Reader role to my user, sqladminuser – but it is unclear where I do this and how. Any hints would be appreciated.

2

Answers


  1. Chosen as BEST ANSWER

    I have found one way to achieve this, though I am not sure if this is the 'best way':

    After the Synapse Workspace is set up, I open Synapse Studio's Develop tab and run below SQL script:

    CREATE DATABASE pqdatalakedb;
    GO
    
    
    USE pqdatalakedb;
    GO
    
    
    CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'XYZStrongPassWordabc123@**1231';
    GO
    
    
    CREATE DATABASE SCOPED CREDENTIAL my_credential
    WITH
        IDENTITY = 'Managed Identity';
    GO
    
    
    CREATE EXTERNAL DATA SOURCE ParquetDataLake
    WITH (
        LOCATION = 'https://<output-storage-account>.dfs.core.windows.net/<output-container>',
        CREDENTIAL = my_credential
    );
    GO
    

    After this, I can authenticate with Synapse within Grafana using the Microsoft SQL Server data source using the user sqlserveradmin and the password I created. For endpoint I use the Synapse serverless SQL endpoint and for database I use pqdatalakedb.

    In my queries I can now use T-SQL in Grafana panels as e.g. below:

    SELECT
       DATEADD(SECOND, DATEDIFF(SECOND, '2000', t) / 5 * 5, '2000') AS time,
       AVG(<signal>) AS avg_signal
    FROM
       OPENROWSET(
             BULK '<deviceid>/<message>/*/*/*/*',
             DATA_SOURCE = 'ParquetDataLake',
             FORMAT = 'PARQUET'
       ) AS r
    WHERE
       CONCAT(r.filepath(1), '-', r.filepath(2), '-', r.filepath(3)) BETWEEN '<yyyy-MM-dd>' AND '<yyyy-MM-dd>'
       AND
       t BETWEEN '<yyyy-MM-dd HH:mm:ss>' AND '<yyyy-MM-dd HH:mm:ss>'
    GROUP BY
       DATEDIFF(SECOND, '2000', t) / 5
    ORDER BY
       time
    OFFSET 0 ROWS
    

    I do feel, however, that there must be a simpler/better way to do this, i.e. by somehow correctly assigning the proper role to my sqlserveradmin user. But I seem unable to figure that route out - any inputs are welcome.

    Note: In my efforts I did seem to manage to somehow provide my sqlserveradmin the right access/role since I did manage to authenticate in Grafana using master as the database and directly refer to the full path in my queries (as illustrated below). But when I create a new Synapse workspace I seem unable to replicate that change, unfortunately, so I am not sure what the correct steps to take are. It seems this would be a cleaner/simpler solution, however:

    ...
    FROM
       OPENROWSET(
             BULK 'https://<output-storage-account>.dfs.core.windows.net/<output-container>/<deviceid>/<message>/*/*/*/*',
             FORMAT = 'PARQUET'
       ) AS r
    ...
    

  2. It seems the issue is that I need to add a `Storage Blob Data Reader` role to my user, `sqladminuser`
    

    As per this MS doc

    the SQL user exists only in serverless SQL pool and permissions are scoped to the objects in serverless SQL pool. Access to securable objects in other services (such as Azure Storage) can’t be granted to SQL user directly since it only exists in scope of serverless SQL pool. The SQL user needs to use one of the supported authorization types to access the files.

    When you want to read external table from serverless pool you can use Active directory authentication. According to this document Mssql server is having active directory authentication:

    enter image description here

    Use active directory admin which is used to create synapse workspace and password to connect with serverless database and you will be able to query external table successfully.

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search