I am new to databricks and currently learning about managed tables. I have created a managed table in databricks and on inspection, it is getting created in below location: dbfs:/user/hive/warehouse/demo.db/race_results_python The microsoft documentation states that this is a root directory.…
I am new to Databricks and would appreciate any help :) I am using Databricks for an ELT project to read from Blob storage transform the data to Dataframes, and write the data to another blob storage(in essence a Lighthouse…
I am trying to read data from mongodb in batch My compute cluster configurations are: Spark Version > 3.5.0 Scala Version > 2.12 I have installed mongodb connector library version is(through maven): org.mongodb.spark:mongo-spark-connector_2.12:10.4.0 I have my schemas defined in a…
Issue Summary: Experiencing problems when transferring data from Oracle Cloud Infrastructure (OCI) to Databricks Delta Lake tables using Qlik Replicate. Source: Oracle Cloud Infrastructure (OCI) Destination: Databricks Delta Lake tables Problem Statement: Data duplicacy Missing data Details: Logs from Qlik…
I have a personal Azure pay-as-you-go subscription in which I have the 'Global Administrator' role. I am also the databricks account administrator. Until two weeks ago, I was able to access the databricks account console without any issues, but I…
I am reaching out to you for assistance with some issues I'm encountering in my Databricks environment. I'm hoping the community can provide some guidance to help me resolve these problems. 1. Error with df.display(): When I try to use…
I have an Azure Data Factory pipeline that runs multiple Databricks Notebooks using job clusters. I want to track down the cost of those job clusters in the Cost Management panel. What I'm interested in is not only the Databricks…
I have a list of .json files that contain person information. One file contains info of one person. I want to load this data into table using pyspark in an Azure Databricks notebook. Let's say the files are built like…
I would like to process the data 24/7 using databricks structured streaming. But before I process all the data I need to apply some transformations on it. therefore I am using foreach. Below is my def from pyspark.sql import DataFrame,…
Error in Azure Databricks Expecting successful output after mounting the container to databricks. I have used wasb for mounting purpose. Error states: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Unable to access container tokyo-olympic-data in account tokyoolympicdatasg.blob.core.windows.net using anonymous credentials, and no credentials found for…