I am trying to test https://aws.amazon.com/blogs/machine-learning/llama-2-foundation-models-from-meta-are-now-available-in-amazon-sagemaker-jumpstart/ llama2 model on sagemaker studio.
I am able to run the code in sagemkaer notebook and when I run it it gives me endpoints a well.
So I have endpoint link. Usually when I use this link with AWS credentials on my postman, it gives me response from the model.
But for Llama 2 model when I am trying to use the endpoint in postman I am getting this error
> {
> "ErrorCode": "CLIENT_ERROR_FROM_MODEL",
> "LogStreamArn": "arn:aws:logs:us-east-1:847137928610:log-group:/aws/sagemaker/Endpoints/meta-textgeneration-llama-2-7b-f-2023-07-26-06-06-21-772",
> "Message": "Received client error (424) from primary with message "{n "code":424,n "message":"prediction failure",n
> "error":"Need to pass custom_attributes='accept_eula=true' as part
> of header. This means you have read and accept the
> end-user-license-agreement (EULA) of the model. EULA can be found in
> model card description or from
> https://ai.meta.com/resources/models-and-libraries/llama-downloads/."n}".
> See
> https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logEventViewer:group=/aws/sagemaker/Endpoints/meta-textgeneration-llama-2-7b-f-2023-07-26-06-06-21-772
> for more information.",
> "OriginalMessage": "{n "code":424,n "message":"prediction failure",n "error":"Need to pass
> custom_attributes='accept_eula=true' as part of header. This means you
> have read and accept the end-user-license-agreement (EULA) of the
> model. EULA can be found in model card description or from
> https://ai.meta.com/resources/models-and-libraries/llama-downloads/."n}",
> "OriginalStatusCode": 424 }
I also tried to invote sagemaker endpoint using Lambda and API Gateways following https://medium.com/@woyera/how-to-use-llama-2-with-an-api-on-aws-to-power-your-ai-apps-3e5f93314b54
But there also I am getting
{
"message": "Internal Server Error"
}
Any suggestion or help recommended here?
2
Answers
Add it as one of headers to your request, since the blog you linked mentions it:
You have to send
custom_attrtibutes
with"accept_eula=true"
in the request headers as follows when you query the deployed model endpoint or the predictor.You can find sample notebook here.