skip to Main Content

I am trying to test https://aws.amazon.com/blogs/machine-learning/llama-2-foundation-models-from-meta-are-now-available-in-amazon-sagemaker-jumpstart/ llama2 model on sagemaker studio.

I am able to run the code in sagemkaer notebook and when I run it it gives me endpoints a well.

So I have endpoint link. Usually when I use this link with AWS credentials on my postman, it gives me response from the model.

But for Llama 2 model when I am trying to use the endpoint in postman I am getting this error

> {
>     "ErrorCode": "CLIENT_ERROR_FROM_MODEL",
>     "LogStreamArn": "arn:aws:logs:us-east-1:847137928610:log-group:/aws/sagemaker/Endpoints/meta-textgeneration-llama-2-7b-f-2023-07-26-06-06-21-772",
>     "Message": "Received client error (424) from primary with message "{n  "code":424,n  "message":"prediction failure",n 
> "error":"Need to pass custom_attributes='accept_eula=true' as part
> of header. This means you have read and accept the
> end-user-license-agreement (EULA) of the model. EULA can be found in
> model card description or from
> https://ai.meta.com/resources/models-and-libraries/llama-downloads/."n}".
> See
> https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logEventViewer:group=/aws/sagemaker/Endpoints/meta-textgeneration-llama-2-7b-f-2023-07-26-06-06-21-772
> for more information.",
>     "OriginalMessage": "{n  "code":424,n  "message":"prediction failure",n  "error":"Need to pass
> custom_attributes='accept_eula=true' as part of header. This means you
> have read and accept the end-user-license-agreement (EULA) of the
> model. EULA can be found in model card description or from
> https://ai.meta.com/resources/models-and-libraries/llama-downloads/."n}",
>     "OriginalStatusCode": 424 }

I also tried to invote sagemaker endpoint using Lambda and API Gateways following https://medium.com/@woyera/how-to-use-llama-2-with-an-api-on-aws-to-power-your-ai-apps-3e5f93314b54

But there also I am getting

{
    "message": "Internal Server Error"
}

Any suggestion or help recommended here?

2

Answers


  1. Add it as one of headers to your request, since the blog you linked mentions it:

    Note that by default, accept_eula is set to false. You need to set
    accept_eula=true to invoke the endpoint successfully. By doing so, you
    accept the user license agreement and acceptable use policy as
    mentioned earlier. You can also download the license agreement.

    Login or Signup to reply.
  2. You have to send custom_attrtibutes with "accept_eula=true" in the request headers as follows when you query the deployed model endpoint or the predictor.

    predictor.predict(payload, custom_attributes="accept_eula=true")
    

    You can find sample notebook here.

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search