What is the best way to explode a comma separated column based on specific conditions? I have some data in the following format: ID col1 1 a100,a101,b100,c100 2 a105,b100 3 b101, c104 what I want to achieve is to: grab…
here is my dataset processed_on|status | ------------+-------+ 2023-01-01 |Success| 2023-01-02 |Success| 2023-01-03 |Success| 2023-01-04 |Fail | 2023-01-05 |Fail | 2023-01-06 |Success| 2023-01-07 |Fail | 2023-01-08 |Success| 2023-01-09 |Success| expected output is -------------------------- start_date|end_date|status -------------------------- 2023-01-01|2023-01-03|Success 2023-01-04|2023-01-05|Fail 2023-01-06|2023-01-06|Success 2023-01-07|023-01-07|Fail 2023-01-08|2023-01-09|Success i…
I have a scala spark code that writes a json file with file name as part-*.json for example (part-00000-14732361-f017-468a-b948-22d3b6d460dc-c000.json). I want to do s3.doesObjectExist(buckey, key) where bucket = xyz and key = abc/def/part-*.json. Looks like s3 doesn't support wildcard search.…
I am trying to read data from a table in a postgresql database and proceed with an ETL project. I have an Docker enviroment using this docker-compose: version: "3.3" services: spark-master: image: docker.io/bitnami/spark:3.3 ports: - "9090:8080" - "7077:7077" volumes: -…
I have a DataFrame with a single column which is a struct type and contains an array. users_tp_df.printSchema() root |-- x: struct (nullable = true) | |-- ActiveDirectoryName: string (nullable = true) | |-- AvailableFrom: string (nullable = true) |…
In our place, we use AWS services for all our data infrastructure and services needs. Our hive tables are external tables and the actual data files are stored in S3. We use Apache Spark for Data ingestion and transformation. We…
I currently have a DynamoDB stream configured which is inputing streams into Kinesis Data streams whenever insertion/updation happens and subsequently I have Glue tables which is taking input from above kinesis stream and then displaying the structural schema and also…
I have newly started working in spark-scala. I have a requirement where in I need to find the sum for few columns within a case statement. I've written the corresponding spark-sql code but unable to implement the same in spark-scala…
I have a json that looks like this: [ { "event_date": "20221207", "user_properties": [ { "key": "user_id", "value": { "set_timestamp_micros": "1670450329209558" } }, { "key": "doc_id", "value": { "set_timestamp_micros": "1670450329209558" } } ] }, { "event_date": "20221208", "user_properties": [ {…
I have a df_movies and col of geners that look like json format. |genres | [{'id': 28, 'name': 'Action'}, {'id': 12, 'name': 'Adventure'}, {'id': 37, 'name': 'Western'}] How can I extract the first field of 'name': val? way #1 df_movies.withColumn…