Question 80625 in Docker Question posted in
The official documentation can be found here.

How to perform GCP authentication from within the Docker image to be run on Cloud Run?

In a Docker image which runs some scripts and at the end of the script I use the bq command to append the data to a BigQuery table. This all works fine offline when I don’t run it within Docker since my local dev environmen has the proper authentication set up.

The issue is how do I set up authentication so that I can run bq from within docker when I do docker run -t my-image.

Also, after I upload it to the gcr.io registry and run it as a job in cloud run. How do I authenticate?

Looking up authentication documentation (e.g. https://cloud.google.com/docs/authentication/provide-credentials-adc) is a like a maze and googling doesn’t turn up very useful results

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    On the GCP CloudRun it's quite simple, in the settings page for the Job you can select a Service Account to run the image as. As long as that service has the right access to bigquery then it should work.

    For offline docker. Still not sure.


  2. 0

    From a recent project a made… (All commands have an equivalent in the GCP Console UI, if you are more familiar with)

    Assumptions:

    1. knowledge of IAM,
    2. the docker container running a service impersonating a dedicated IAM service account (SA).

    Create the service account (SA) and deploy the docker image as a Run service ————–

    # RUN_SERVICE_NAME the name of your Run service as you deploy it in GCP
# SERVICE_ACCOUNT the name of the dedicated service account the service uses
# SERVICE_ACCOUNT_FRIENDLY_NAME an easy name for you to read
# SERVICE_ACCOUNT_EMAIL the email GCP gives you back when you create the SA

gcloud iam service-accounts create $SERVICE_ACCOUNT --display-name "$SERVICE_ACCOUNT_FRIENDLY_NAME"

gcloud run deploy $RUN_SERVICE_NAME 
    --service-account $SERVICE_ACCOUNT_EMAIL
    ....
    ....

    The SA must have roles to access or edit the BQ dataset and table (ref.: https://cloud.google.com/bigquery/docs/control-access-to-resources-iam#grant_access_to_a_table_or_view)

    You can download BQ config to a json file your dev machine…

    For Dataset ———————————–

    # BigQuery get the dataset full info
bq show 
   --format=prettyjson 
   $PROJECT:$DATASET_NAME > $DATASET_NAME.json

    Then update the file to include the BQ roles.. (<SERVICE_ACCOUNT_EMAIL> replace with your SA email, <BQ_ROLE_1> and <BQ_ROLE_2> with your roles).:

    {
  "access": [
    ....
    ....

    {
      "role": "<BQ_ROLE_1>",
      "userByEmail": "<SERVICE_ACCOUNT_EMAIL>"
    }
  ],

  ....
  ....
}

    Finally update the policy:

    # BigQuery update dataset permissions
bq update --source $DATASET_NAME.json $PROJECT:$DATASET_NAME

    For Table ——————————–

    # BigQuery get the policy of a table
bq get-iam-policy 
   $PROJECT:$DATASET_NAME.$DATASET_TABLE_NAME 
   > $DATASET_NAME.$DATASET_TABLE_NAME.json

    Then update the file to include the BQ roles, eg.:

    {
  "bindings": [
    {
      "members": [
        "<SERVICE_ACCOUNT_EMAIL>"
      ],
      "role": "<BQ_ROLE_2>"
    }
  ],
  ....
  ....
}

    Finally update the policy:

    bq set-iam-policy $PROJECT:$DATASET_NAME.$DATASET_TABLE_NAME $DATASET_NAME.$DATASET_TABLE_NAME.json

    In your service code (example made with NodeJS):

    const { BigQuery } = require('@google-cloud/bigquery');
const bigqueryClient = new BigQuery();
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search