apache-spark Questions

Json Column in delta table databricks

June 13, 2023
Amdbi
2 Answers

I have a view in databricks and in one column the content is a json, example: [{"ECID":100017056,"FIRST_NAME":"Ioannis","LAST_NAME":"CHATZIZYRLIS","TITLE":"Mr","GENDER":"M","DATE_OF_BIRTH":"1995-04-14","PLACE_OF_BIRTH":"Greece","COUNTRY_OF_BIRTH":"GR","NATIONALITY":"GR","RESIDENCE":"GR","ADDRESS":[{"TYPE":1,"STREET_1":"Gizi 6","CITY":"Agios Doe","POSTAL_CODE":"111 34","COUNTRY":"GR","MOBILE":"0000000","EMAIL":" "}]}] I want to retrieve the content of that column so that I can post it to an api. I…

VIEW QUESTION

Amazon web services – Write pyspark dataframe into specific number of parquet files in total across all partition columns

June 13, 2023
haneulkim
2 Answers

Using PySpark 2.4.7 My goal is to write PySpark DataFrame into specific number of parquet files into AWS S3. Say I want to write PySpark DataFrame into 10 parquet files. This is my approach df.repartition(10).write.mode("append").partitionBy(partition_cols).saveAsTable(<db.table>) This writes 10 parquet files…

VIEW QUESTION

Amazon web services – How to access existing Glue catalog from EMR?

June 13, 2023
Prabodh Mhalgi
2 Answers

I have created a Glue catalog in my account. It has 1 DB and 1 table. I followed this guide from AWS and created my EMR cluster. However, when I run spark-shell and try to access Glue catalog, I am…

VIEW QUESTION

Ubuntu – How to read the correct values from Parquet files?

June 4, 2023
burak
2 Answers

I want to read the Parquet files I get from the readSteam() method from a Kafka topic. However, when I read these Parquet files, I get a dataframe that I don't want instead of real values. import glob parquet_files =…

VIEW QUESTION

Can Scala read JSON with string values without quotes?

May 31, 2023
Ankur
2 Answers

I have a log in below format(json/ion) where the value of time_of_initial_request is in timestamp format but without double quotes. {post_handler_type:"trace", default_marketplace:"prod_iad", request_id:"5MTB3656X0JRL38R6LJY", rest_uri:"productApi/feature", client_logging_id:"PredictionMetadata", time_of_initial_request:2023-05-24T15:00:00.577Z } Since the field(key) name does not have quotes, I am using the allowUnquotedFieldNames…

VIEW QUESTION

Can pyspark read data from multiple mongodb collections?

May 25, 2023
yuser099881232
2 Answers

I have a mongo db with multiple collections (let's say 3). I want to join them all so that I can do some aggregations. I was able to connect to a local db and one collection using documentation found oneline…

VIEW QUESTION

Spark handle json with dynamically named subschema

May 23, 2023
Jonathan M
2 Answers

I have a JSON file in the following structure: { "10712": { "id": "10712", "age": 27, "gender": "male" }, "217": { "id": "217", "age": 60, "gender": "female" } } This causes a problem when importing using spark.read.json, because of the…

VIEW QUESTION

Explode a json string present in pyspark dataframe

April 25, 2023
Ben
2 Answers

I have a JSON string substitutions as a column in dataframe which has multiple array elements that I want to explode and create a new row for each element present in that array. There are other columns present in the…

VIEW QUESTION

Does JDBC for Postgres have the equivalent of DirectJoin for Cassandra? – Postgresql

April 23, 2023
Danilo Rodrigues
2 Answers

With Cassandra, if I need some specific data, all I need to do is pass the keys on join columns (with DirectJoin enabled). This works very well because is fast. But if I do a join with JDBC, first spark…

VIEW QUESTION

Azure – Decrypt an encrypted string in Databricks using GCM mode

April 20, 2023
sm925
2 Answers

I'm trying to decrypt a string in databricks using aes_decrypt function with GCM mode. But, it doesn't seem to work with IV. Didn't find any details about how to use IV with aes_decrypt function : https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/functions/aes_decrypt?source=recommendations Here are the details…

VIEW QUESTION