I have a Spark Dataframe which contains in every single row two items: a file name (with an extension, like for instance .jpg), and the content of the file in bytes. I would like to write a process that takes…
We want to enable the access for the users to create token. Now, I have logged in to the databricks account using the admin access, and I go to "Access Control" -> "Personal Access Tokens" and I want to enable…
In Pyspark, I have 2 dataframe. 1st dataframe say df1 is empty dataframe created from a schema. 2nd dataframe df2 is non-empty dataframe filled from a csv file. Now I want to merge such that all below scenarios are covered.…
I'm trying to filter out data from the current date to last 3 years and trying to use this for Pyspark dataframe. below sql query needs to convert into Pyspark dataframe format date >= dateadd(month,-4,current_date) How to write above sql…
I have made a project tree: . ├── calculations │ ├── calculation1.py │ └── calculation2.py | └── __init__.py ├── final_combinations │ ├── project.py │ └── non_project.py In project.py I import the calculations from calculations folder: from calculations.calculation1 import calculation1 from…
I am building a data lake on Databricks. Basically, it consists of a pipeline of Databricks jobs, written in Python (and using PySpark for some). Now, I was wondering how could I test those notebooks. Integration test As I have…
I have tried to create a persistent view with a simple select statement that just has a word with greek characters. CREATE OR REPLACE VIEW misc.XX AS SELECT 'ΤΕΣΤ' If I try to select that view, I get ???? instead…
I have a function that takes a dataframe as a parameter and calculates the NULL value counts and NULL value percentage and returns a dataframe with column_name, null value count and null percentages. How can I register it as UDF…
Question: I'm trying to set up a MongoDB instance in databricks notebook using Databricks' DBFS mounted storage as its data storage location. I can mount my storage (azure blob or s3 bucket) to DBFS and can access it without issues…
I'm using a notebook on Azure Databricks, this notebook is in my user repo. I want to write a csv file created by this notebook in this repo. When i'm using the code below : df_pandas.to_csv('test.csv', index=False, header=False) There is…