Question 311318 in PostgreSQL Question posted in
The official documentation can be found here.

PostgreSQL to Google BigQuery to Cloud function real time synchronisation

I’m currently utilizing Datastream to seamlessly replicate real-time changes in data from PostgreSQL to BigQuery, and the process is functioning smoothly. However, I find myself in need of a solution for triggering a Cloud Function immediately after any updates or insertions from PostgreSQL to BigQuery.

I would greatly appreciate any suggestions on the most effective way to achieve this.

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    Thank you for your update. I encountered an issue while using google.cloud.bigquery.v2.JobService.InsertJob. The problem arises when employing a Cloud Function with a join query, resulting in continuous triggers. For each insert, there are 3 to 5 redundant audit log entries, leading to multiple triggers for a single insert.

    Although the appendrow functionality works, it introduces a delay of 30 seconds to 4 minutes before triggering the function.

    I'm seeking guidance on how to ensure the insert job triggers the Cloud Function only once, or how to minimize the delay in the appendrow method.


  2. 0

    You can’t trigger a Cloud Functions directly from Datastream but you can however achieve the similar behaviour by using 2nd generation Cloud Functions and an Eventarc trigger, in your case of type google.cloud.bigquery.v2.JobService.InsertJob. Full list of Eventarc event types can be found here

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search