Question 388718 in Azure Question posted in
The official documentation to get you started can be found here.

How to deal with wasb error in mounting azure databricks with containers?

Error in Azure Databricks

Expecting successful output after mounting the container to databricks. I have used wasb for mounting purpose.
Error states: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Unable to access container tokyo-olympic-data in account tokyoolympicdatasg.blob.core.windows.net using anonymous credentials, and no credentials found for them in the configuration.

Couldn’t figure out the error as my authentication is absolute.

Tags:

2

Answers


  1. 0

    In your config you declare the provider – "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider".

    So you need to start your source with "abfss://" and not "wasbs://"

    abfss – is for Azure data lake storage

    wasbs – is for accessing blob storage

    Login or Signup to reply.
  2. 0

    Whenever you need to read from or store files in Blob Storage or ADLS Gen2, it is mandatory to mount it using either a SAS token or an Account Access Key.

    Below is the syntax/code for Blob Storage.

    Step 1: Blob Storage account Access Key or SAS Token.

    enter image description here

    I have tried the following approach:

    storageAcc = "dileepstgacc"
storageAccessKey = "D1BD/sSgllh2rdLxN5RKQQ873GMh64ecD4MNTyAty61/3X3QR7XgcriXfdNCycggCbt/4BmUMXni+ASto8s28A=="
#sasToken = <sas-token>
Container = "folder02"
mountPoint = "/mnt/new"
if any(mount.mountPoint == mountPoint for mount in dbutils.fs.mounts()):
  print("mountPoint--> ",mountPoint)
  dbutils.fs.unmount(mountPoint)
  print("End of Unmount")
try:
  dbutils.fs.mount(
    source = "wasbs://{}@{}.blob.core.windows.net".format(Container, storageAcc),
    mount_point = mountPoint,
    #extra_configs = {'fs.azure.account.key.' + storageAcc + '.blob.core.windows.net': storageAccessKey}
    extra_configs = {'fs.azure.account.key.{0}.blob.core.windows.net'.format(storageAcc): storageAccessKey}
    #extra_configs = {"fs.azure.sas.{0}.blob.core.windows.net".format(Container):sasToken}
    #extra_configs = {'fs.azure.sas.' + Container + '.' + storageAcc + '.blob.core.windows.net': sasToken}
  )
  print("mount succeeded!")
except Exception as e:
  print("mount exception", e)

    Results:

    enter image description here
    If you want to mount ADLS Gen2 instead of Blob Storage, you only need to make a very minor change in the same code, which is highlighted below in bold.

    For Blob Storage: wasbs://@.blob.core.windows.net/

    For ADLS Gen2: abfss://@.dfs.core.windows.net/

    The full form of wasbs is Windows Azure Storage Blob.

    The full form of abfs is Azure Blob File System.

    To check all mounted paths, use the command below:

    dbutils.fs.mounts()

    To unmount:

    dbutils.fs.unmount("/mnt/new")

    Reference:
    Mount Blob Storage and ADLS Gen2 in Azure Databricks

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search