Amazon Web Services Question posted in
The official Amazon Web Services documentation can be found here.

Amazon web services – How to pass system prompt to claude 3 haiku on AWS Bedrock Using JavaScript SDK V3

I can’t figure out the correct way to pass a system prompt to claude 3 haiku through AWS Bedrock InvokeModelWithResponseStreamCommand. When I try to add { role: "system", content: systemPrompt } to the messages array it breaks. I’ve scoured Anthropic’s and AWS Bedrock’s documentation but can’t find out whether system prompts are suppposed to be in a separate field from messages. According to anthropic docs it should be, but it does not appear that AWS Bedrock supports a separate system prompt field outside of messages in its JS SDK.

import { NextResponse } from "next/server";
import { BedrockRuntimeClient, InvokeModelWithResponseStreamCommand } from "@aws-sdk/client-bedrock-runtime";

const systemPrompt = "todo"

export async function POST(req) {

  // Create a Bedrock Runtime client in the AWS Region you want to use.
  const client = new BedrockRuntimeClient({ region: "us-east-1" });
  const data = await req.json();
  // Set the model ID, e.g., Claude 3 Haiku.
  const modelId = "anthropic.claude-3-haiku-20240307-v1:0";
  console.log(data)
  const payload = {
    anthropic_version: "bedrock-2023-05-31",
    max_tokens: 10000,
    messages: [
      {"role": "user", "content": "Hello there."},
      {"role": "assistant", "content": "Hi, I'm Claude. How can I help you?"},
      {"role": "user", "content": "Can you explain LLMs in plain English?"},
    ],
  }

  // Create a command with the model ID, the message, and a basic configuration.
  const command = new InvokeModelWithResponseStreamCommand({
    contentType: "application/json",
    body: JSON.stringify(payload),
    modelId,
  });
  
  const stream = new ReadableStream({
    async start(controller) {
      try {
        // Send the command to the model and wait for the response
        const response = await client.send(command);
        let completeMessage = "";
        // Extract and print the streamed response text in real-time.
        for await (const item of response.body) {
          const chunk = JSON.parse(new TextDecoder().decode(item.chunk.bytes));
          const chunk_type = chunk.type;
      
          if (chunk_type === "content_block_delta") {
            const text = chunk.delta.text;
            completeMessage = completeMessage + text;
            controller.enqueue(text);
            process.stdout.write(text);
          }
        }
      } catch (err) {
        controller.error(err)
      } finally {
        controller.close()
      }
    }
  })

  return new NextResponse(stream);
}

The error when I try to add the system prompt to the messages array:

Error: failed to pipe response
    ...
  [cause]: ValidationException: Malformed input request: #: subject must not be valid against schema {"required":["messages"]}#/messages/0/role: system is not a valid enum value, please reformat your input and try again.
      at de_ValidationExceptionRes (webpack-internal:///(rsc)/./node_modules/@aws-sdk/client-bedrock-runtime/dist-es/protocols/Aws_restJson1.js:379:23)
      at de_CommandError (webpack-internal:///(rsc)/./node_modules/@aws-sdk/client-bedrock-runtime/dist-es/protocols/Aws_restJson1.js:212:25)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async eval (webpack-internal:///(rsc)/./node_modules/@smithy/middleware-serde/dist-es/deserializerMiddleware.js:8:24)
      at async eval (webpack-internal:///(rsc)/./node_modules/@smithy/core/dist-es/middleware-http-signing/httpSigningMiddleware.js:25:20)
      at async eval (webpack-internal:///(rsc)/./node_modules/@smithy/middleware-retry/dist-es/retryMiddleware.js:41:46)
      at async eval (webpack-internal:///(rsc)/./node_modules/@aws-sdk/middleware-logger/dist-es/loggerMiddleware.js:9:26)
      at async Object.start (webpack-internal:///(rsc)/./app/api/chat/route.js:118:34) {
    '$fault': 'client',
    '$metadata': {
      httpStatusCode: 400,
      requestId: '5c49e0b0-d781-4c5a-bd7d-0823a50a763f',
      extendedRequestId: undefined,
      cfId: undefined,
      attempts: 1,
      totalRetryDelay: 0
    }
  }
}
 POST /api/chat 500 in 928ms

I’ve tried to add { role: "system", content: systemPrompt } as an additional message in the messages array and as a separate ("system") field. Using the external system field approach appears to be the correct way according to Anthropic’s docs, but AWS Bedrock doesn’t appear to support an external system field in its InvokeModelWithResponseStreamCommand Bedrock Runtime API. I’m expecting a streaming response that takes in user prompts and responds within the constraints of the system prompt.
https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModelWithResponseStream.html
https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/system-prompts
https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/javascript_bedrock-runtime_code_examples.html#anthropic_claude

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    Looks like max_tokens can't be more than 4096 and the correct way is to pass in the prompt as a separate system field: https://github.com/aws/aws-sdk-js-v3/issues/5853
    https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-anthropic-claude-messages.html#model-parameters-anthropic-claude-messages-request-response
    The code below works in my experience:

    const payload = {
    anthropic_version: "bedrock-2023-05-31",
    max_tokens: 4096,
    system: systemPrompt,
    messages: [
      {"role": "user", "content": "Hello there."},
      {"role": "assistant", "content": ""},
    ],
  }

  2. 0

    I had exactly this issue and finally managed to get it working by tweaking some rather cryptic parameters. A better solution, the one I settled on and is recommended by AWS is to use the ConverseCommand API rather than the InvokeModelCommand API. The Converse API standardises the parameters between models.

    const client = new BedrockRuntimeClient({region: "us-west-2"});

//const model = "meta.llama3-8b-instruct-v1:0";
const model = "anthropic.claude-3-haiku-20240307-v1:0";
const system = `
Extract the core message from the email, removing any forwarding headers, email signatures, and other extraneous content.
Focus on retaining only the essential information related to the message's purpose.
Retain line breaks in the output.
Remove any signoff details at the bottom of the email such as thank you's or names and contact details.
Remove any email signatures.
`;

export async function cleanEmail(email: string) {
    const command = new ConverseCommand({
        modelId: model,
        system: [{ text: system }],
        messages: [{
            role: "user",
            content: [{ text: email }],
        }],
        inferenceConfig: { temperature: 0 },
    });

    const response = await client.send(command);
    if (response.output?.message?.content) {
        console.log(JSON.stringify(response, null, 2));
        return response.output.message.content[0].text;
    }
    return email;
}
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search