databricks Questions

Json Column in delta table databricks

June 13, 2023
Amdbi
2 Answers

I have a view in databricks and in one column the content is a json, example: [{"ECID":100017056,"FIRST_NAME":"Ioannis","LAST_NAME":"CHATZIZYRLIS","TITLE":"Mr","GENDER":"M","DATE_OF_BIRTH":"1995-04-14","PLACE_OF_BIRTH":"Greece","COUNTRY_OF_BIRTH":"GR","NATIONALITY":"GR","RESIDENCE":"GR","ADDRESS":[{"TYPE":1,"STREET_1":"Gizi 6","CITY":"Agios Doe","POSTAL_CODE":"111 34","COUNTRY":"GR","MOBILE":"0000000","EMAIL":" "}]}] I want to retrieve the content of that column so that I can post it to an api. I…

VIEW QUESTION

Can I read a Json from a Spark Dataframe Column w/o RDD/collect()?

May 18, 2023
noobCoder50
2 Answers

I have a dataframe that looks like this, +-------------------------------------------+ | output| +-------------------------------------------+ |{"COLUMN1": "123", "COUMN2": {"A":1 "B":2}}| +-------------------------------------------+ And i just want to read the json as a string or dictionary in a variable so that i could do further…

VIEW QUESTION

Can Azure search for specific values in a column and its corresponding data from other columns?

May 16, 2023
Arun
2 Answers

I have following scenerio where I have to check for values in a column and its corresponding values in other columns. If any one of the value is not null then need to return true. Suppose I have to check…

VIEW QUESTION

Mounted Azure Storage shows mount.err inside folder while reading from Azure Databricks

May 4, 2023
Vishnu Suresh
2 Answers

I'm using Azure Databricks notebook to read a excel file from a folder inside a mounted Azure blob storage. The mounted excel location is like : "/mnt/2023-project/dashboard/ext/Marks.xlsx".2023-project is the mount point and dashboard is the name of the container. When…

VIEW QUESTION

Evaluate a date format from a String value – Mysql

April 28, 2023
sayan nandi
5 Answers

I have a requirement where a filed value is "CAH132242F0A" from this i have to come up with a date. so here "13" is the year and so yyyy = "2013" and "224" th day. so i have to extract…

VIEW QUESTION

Azure Databricks: Getting occurred ERROR `CrossTenantUserAssignmentRequestForbidden` when starting compute cluster

April 28, 2023
Yangwook Ian Jeong
2 Answers

The Subscription itself, which includes Azure Databricks created in Directory A of account A, has been transferred to directory B of account B. A Compute cluster and Workflow were already created in directory A of account A. It for running…

VIEW QUESTION

How to migrate your dbt project from dbt-postgres to dbt-databricks – Postgresql

April 21, 2023
Derik Roby
2 Answers

I have my my dbt-project configured and running for dbt-postgres. Is there an easy way to port it to dbt- databricks without having to rewrite the models. Context: I am building a dbt demo on postgres, databricks and Azure SQL…

VIEW QUESTION

Azure – Decrypt an encrypted string in Databricks using GCM mode

April 20, 2023
sm925
2 Answers

I'm trying to decrypt a string in databricks using aes_decrypt function with GCM mode. But, it doesn't seem to work with IV. Didn't find any details about how to use IV with aes_decrypt function : https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/functions/aes_decrypt?source=recommendations Here are the details…

VIEW QUESTION

Azure – How to extract Sheet names from Excel file using "com.crealytics.spark.excel" in Databricks (PySpark)

April 5, 2023
AzSurya Teja
2 Answers

I have an Excel file in the azure datalake ,I have read the excel file like the following ddff=spark.read.format("com.crealytics.spark.excel") .option("header", "true") .option("sheetName","__all__") .option("inferSchema","true") .load("abfss://[email protected]/file.xlsx") Now Iam confused how to get just the sheetnames from that Excel file,is there any direct…

VIEW QUESTION

Azure – Scala recursive dbutils.fs.ls

March 23, 2023
Diego Eick Moreira
2 Answers

I have the following scala code in Azure Databricks that creates a dataframe with the list of files and file properties for an specific folder. Can someone help me to transform that code to recursively navigate to each sub folder…

VIEW QUESTION