Javascript Question posted in
A very good W3school tutorial can be found here.

Javascript – Permission Error When Executing Vertex AI Fine Tuning Job with Node.js: Service Account Roles Insufficient?

I’m using a pipeline job to fine tune a text-bison model with Vertex AI on GCP. My API is logged using a dedicated service account. It has the roles : Vertex AI Service Agent and Service Accounts User.

I’m using this Node.js code :

import aiplatform from '@google-cloud/aiplatform';

class ModelTuningController {
  initiateModelTuning = catchAsync(async (req, res) => {
    const { modelDisplayName = 'fineTunedTest' } = req.body;

    const { PipelineServiceClient } = aiplatform.v1;

    const location = 'us-central1';
    const project = 'projectID';
    const model = 'text-bison@001';

const pipelineClient = new PipelineServiceClient({
  apiEndpoint: `${location}-aiplatform.googleapis.com`,
  keyFilename: config.vector_store_key_filename,
});

const parameters = {
  source_model: { stringValue: model },
  train_dataset: {
    stringValue: 'gs://path/file.jsonl',
  },
  tuned_model_display_name: {
    stringValue: modelDisplayName,
  },
  epochs: {
    numberValue: 4,
  },
  learning_rate_multiplier: {
    numberValue: 1,
  },
};

const runtimeConfig = {
  gcsOutputDirectory: 'gs://output-path',
  parameterValues: parameters,
};

try {
  const parent = `projects/${project}/locations/${location}`;
  const pipelineJob = {
    displayName: modelDisplayName,
    runtimeConfig,
  };

  const request = {
    parent,
    pipelineJob,
  };

  const [response] = await pipelineClient.createPipelineJob(request);

  return sendResponse(res, {
    message: ModelTuningMessages.jobInitiated(response.name),
  });
} catch (err) {
  return sendError(
    err,
    new ApiError(
      httpStatus.INTERNAL_SERVER_ERROR,
      ModelTuningMessages.jobError


       )
      );
    }
  });
}

export default ModelTuningController;

I have the following error : You do not have permission to act as service_account: ${projectID}[email protected]. (or it may not exist).

The problem is ${projectID}[email protected] is the project default service account. My guess it that my service account should act as the default service account of the project.

Does my service account is missing a role to execute the fine tuning job on Vertex AI ?

___________

EDIT

It seems like I needed to activate the Compute Engine API to have the default compute engine service account to make it work !

However I still have the error : Error: 3 INVALID_ARGUMENT: . It’s hard to know which argument is invalid. This is the request JSON I send to the pipeline job :

{
  "parent": "projects/${projectID}/locations/us-central1",
  "pipelineJob": {
    "displayName": "fineTunedTest",
    "runtimeConfig": {
      "gcsOutputDirectory": "gs://${output}",
      "parameterValues": {
        "source_model": { "stringValue": "gemini-1.0-pro-002" },
        "train_dataset": { "stringValue": "gs://${input}/${file}" },
        "tuned_model_display_name": { "stringValue": "fineTunedTest" },
        "epochs": { "numberValue": 4 },
        "learning_rate_multiplier": { "numberValue": 1 }
      }
    }
  }
}
Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    For the first problem : (You do not have permission to act as service_account: ${projectID}[email protected]. (or it may not exist).) I needed to activate the Compute Engine API to have the default compute engine service account. My service account only needed the roles Vertex AI Service Agent and Service Accounts User.

    For the second problem : (Error: 3 INVALID_ARGUMENT:) I needed to add the missing parameter templateUri to my pipelineJob object. His value is https://us-kfp.pkg.dev/ml-pipeline/large-language-model-pipelines/tune-large-model/v2.0.0 for text model and https://us-kfp.pkg.dev/ml-pipeline/large-language-model-pipelines/tune-large-chat-model/v3.0.0 for chat model. See this for more informations.

    My updated code is :

    import aiplatform from '@google-cloud/aiplatform';

class ModelTuningController {
  initiateModelTuning = catchAsync(async (req, res) => {
    const { modelDisplayName = 'fineTunedTest' } = req.body;

    const { PipelineServiceClient } = aiplatform.v1;

    const location = 'us-central1';
    const project = 'projectID';
    const model = 'text-bison@001';

const pipelineClient = new PipelineServiceClient({
  apiEndpoint: `${location}-aiplatform.googleapis.com`,
  keyFilename: config.vector_store_key_filename,
});

const parameters = {
  source_model: { stringValue: model },
  train_dataset: {
    stringValue: 'gs://path/file.jsonl',
  },
  tuned_model_display_name: {
    stringValue: modelDisplayName,
  },
  epochs: {
    numberValue: 4,
  },
  learning_rate_multiplier: {
    numberValue: 1,
  },
};

const runtimeConfig = {
  gcsOutputDirectory: 'gs://output-path',
  parameterValues: parameters,
};

try {
  const parent = `projects/${project}/locations/${location}`;
  const pipelineJob = {
    displayName: modelDisplayName,
    templateUri: 'https://us-kfp.pkg.dev/ml-pipeline/large-language-model-pipelines/tune-large-model/v2.0.0',
    runtimeConfig,
  };

  const request = {
    parent,
    pipelineJob,
  };

  const [response] = await pipelineClient.createPipelineJob(request);

  return sendResponse(res, {
    message: ModelTuningMessages.jobInitiated(response.name),
  });
} catch (err) {
  return sendError(
    err,
    new ApiError(
      httpStatus.INTERNAL_SERVER_ERROR,
      ModelTuningMessages.jobError
       )
      );
    }
  });
    }

export default ModelTuningController;

  2. 0

    The issue is not passing proper service account credential and granting proper IAM permission. There are two solutions (the second is preferred):

    1. Grant additional permission to the the default compute service account.
      Vertex AI pipeline will fallback to use default compute service account ${projectID}[email protected] when you don’t specify a dedicated custom service account. To run Vertex pipeline, the default service account needs to have the following IAM permissions:
    • Vertex AI User (I think you are missing this one)
    • Service Account User
    • Storage Object Admin (optional when the pipeline needs to interact with storage bucket)
      Try granting vertex AI user role to the default service account.
    1. Use a dedicated custom service account. You can do this with GCP Application Default Credentials or pass credential to the API. Following is a sample snippet of passing your custom SA:
    import aiplatform from '@google-cloud/aiplatform';
import { Auth, google } from "googleapis";

class ModelTuningController {
initiateModelTuning = catchAsync(async (req, res) => {
  const { modelDisplayName = 'fineTunedTest' } = req.body;

  const { PipelineServiceClient } = aiplatform.v1;
  # set up auth
  // Path to your service account key file
  const keyFilename = '/path/to/your/service-account.json';

  // Create a new GoogleAuth instance with the service account key file
  const auth = new Auth.GoogleAuth({
   keyFile: keyFilename,
   scopes: 'https://www.googleapis.com/auth/cloud-platform'
  });

 // Get credentials
 const client = await auth.getClient();

 const clientOptions = {
  apiEndpoint: '${location}-aiplatform.googleapis.com',
  auth: client // Pass the authenticated client object here
 };

  const location = 'us-central1';
  const project = 'projectID';
  const model = 'text-bison@001';

  const pipelineClient = new PipelineServiceClient(clientOptions);
  ...

    Keep in mind, the preferred authentication method depends on where and how you run the pipeline Node.js code. i.e running locally vs. running in GCP cloud function. Ideally, you don’t want to export the service account key file for better security.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search