I'm using the jupyter/pyspark-notebook docker image to develop a spark script. My dockerfile looks like this: FROM jupyter/pyspark-notebook USER root COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt && rm requirements.txt # this is a default user and the…
I'm working with a Delta table in Databricks that is partitioned by a Year column, and I have a timestamp column in the table. I want to optimize my queries to fetch data within a specific timestamp range (e.g., between…
I run my AWS Glue jobs locally in a docker container (AWS Glue lib 4.0) and want to convert/write a pandas dataframe to delta format. I added spark = SparkSession.builder .appName("YourAppName") .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") .getOrCreate() sys.argv += ['--datalake-formats', 'delta']…
I have a view in databricks and in one column the content is a json, example: [{"ECID":100017056,"FIRST_NAME":"Ioannis","LAST_NAME":"CHATZIZYRLIS","TITLE":"Mr","GENDER":"M","DATE_OF_BIRTH":"1995-04-14","PLACE_OF_BIRTH":"Greece","COUNTRY_OF_BIRTH":"GR","NATIONALITY":"GR","RESIDENCE":"GR","ADDRESS":[{"TYPE":1,"STREET_1":"Gizi 6","CITY":"Agios Doe","POSTAL_CODE":"111 34","COUNTRY":"GR","MOBILE":"0000000","EMAIL":" "}]}] I want to retrieve the content of that column so that I can post it to an api. I…
i was able to get row values from delta table using foreachWriter in spark-shell and cmd but while writing the same code in azure databricks it doesn't work. val process_deltatable=read_deltatable.writeStream.foreach(new ForeachWriter[Row] { def process(value: Row):Unit ={ val Telemetery= value.getString(0) println(Telemetery)…
I am still new to Redshift service and quite confused of when to use or what data to put into Spectrum. Suppose I have star schema data warehouse on Redshift, should I put fact table or dim table into Spectrum(external…
I am trying to run the new apache pulsar Lakehouse Sink Connector and I am getting java.lang.IllegalArgumentException Below is my setup. docker-compose.yaml file: version: '3.7' volumes: mssql-data: minio-data: networks: oentity: driver: bridge services: pulsar: image: apachepulsar/pulsar:latest command: bin/pulsar standalone hostname:…
We have our delta-lake from which we can query data. We also connected it with Power BI to make interactive dashboards. It's in production. Now we want to use Azure Purview to avail all the data governance and data catalog…