I'm working with a Delta table in Databricks that is partitioned by a Year column, and I have a timestamp column in the table. I want to optimize my queries to fetch data within a specific timestamp range (e.g., between…
The following bicep deployment has suddenly started failing and I do not understand why var managedResourceGroupName = 'rg-metadata-${dataBricksName}' resource databrickworkspace 'Microsoft.Databricks/workspaces@2018-04-01' = { name: dataBrickName location: location sku: { name: dataBrickPricingTier } properties: { managedResourceGroupId: managedResourceGroup.id parameters: { enableNoPublicIp: {…
I am using databricks for a specific workload. This workload involves an approx of 10 to 200 dataframes that are read and written to a storage location. This workload can benefit from parallelism. The constraint i have is cost optimization.…
I am using the Databricks VSCode extension for development in an IDE. The basic functionalities are all working well. I connected to an Azure Databricks workspace with Unity Catalog enabled, selected an active cluster (DBR 13.2) and configured the sync…
I have these following inputs. 1.Endpoint 2.Database 3.Collections Based on these inputs needs to establish a connection with Azure Cosmos Db from databricks Tried this approach: installed importted com.azure.cosmos.spark maven package in databricks cluster with the DBR of 11.3LTS and…
I have an array type column subscription in my table with columns Sample Table +-------+--------+----+----------+ |user_id| Subscription|Activity | +-------+-------------+----------+ |111 |[A,C,B] |20 | |-------+-------------+----------+ |222 |[C,A] |10 | |-------+-------------+----------+ |333 |[B] | 5 | +-------+-------------+----------+ I want to change my…
Question: When doing a Purview scan on Azure Databricks, I get the following error. What could be the cause of the error and how we can fix it? Remarks: For safety, I have replaced the 16 digits in the error…
I have a view in databricks and in one column the content is a json, example: [{"ECID":100017056,"FIRST_NAME":"Ioannis","LAST_NAME":"CHATZIZYRLIS","TITLE":"Mr","GENDER":"M","DATE_OF_BIRTH":"1995-04-14","PLACE_OF_BIRTH":"Greece","COUNTRY_OF_BIRTH":"GR","NATIONALITY":"GR","RESIDENCE":"GR","ADDRESS":[{"TYPE":1,"STREET_1":"Gizi 6","CITY":"Agios Doe","POSTAL_CODE":"111 34","COUNTRY":"GR","MOBILE":"0000000","EMAIL":" "}]}] I want to retrieve the content of that column so that I can post it to an api. I…
I have a dataframe that looks like this, +-------------------------------------------+ | output| +-------------------------------------------+ |{"COLUMN1": "123", "COUMN2": {"A":1 "B":2}}| +-------------------------------------------+ And i just want to read the json as a string or dictionary in a variable so that i could do further…
I have following scenerio where I have to check for values in a column and its corresponding values in other columns. If any one of the value is not null then need to return true. Suppose I have to check…