Question 115123 in PostgreSQL Question posted in
The official documentation can be found here.

Postgresql – SQL vs PySpark/Spark SQL

Could someone please help me understand why we need to use PySpark or SprakSQL etc if the source and target of my data is the same DB?

For example, lets say I need to load data to table X in PostgresDB from tables X and Y. Would it not be simpler and faster to just do it in Postgres instead of using SprakSQL or PySpark etc?

I understand the need for these solutions if data is from multiple sources, but if it is from same source, do I need to use PySpark?

Tags:

3

Answers


  1. Chosen as BEST ANSWER
    0

    Thank you all for the feedback. I think I will use glue pyspark if source and destination are different. Else i will use glue python with jdbc connection and have one session do the tasks without bringing data to dataframes.


  2. 0

    You can use spark when you want to do heavy data transformations, it makes it easier to load and process due to distributed processing.

    It totally depends on how large is the data and how you want to transform it.

    Using Postgres will be a good idea if data is relatively small and no transformation is required.

    Login or Signup to reply.
  3. 0

    It is not necessary to use PySpark. Both PySpark & SparkSQL have their value in managing/manipulating large volumes of data few hundred of GBs, TBs, or PBs in a distributed computing setup. If this is your case, please use PySpark, it will be more efficient to load, manipulate, process/shape the data before inserting it into another table.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search