I have deployed Llama 3.1 70B and Llama 3.1 8B on my system and it works perfectly for the 8B model. When I tested it for 70B, it underutilized the GPU and took a lot of time to respond. Here…
I want to run Ollama docker image. This exposes entire API with its endpoints. I want to secure it by adding reverse proxy on top of it and i have chosen Nginx for the job. So I need to add…
I am running meta-llama/Meta-Llama-3-8B-Instruct endpoint on AWS and for some reason cannot get reasonable output when prompting the model. It hallucinates even when I send through a simple prompt. Could someone please advise what I am doing wrong? Exemplary prompt:…
I'm running pytorch on an ubuntu server 18.04 LTS. I have an nvidia gpu with 8 GB or ram. I'd like to experiment with the new llma2-70B-chat model. I'm trying to use peft and bitsandbytes to reduce the hardware requirements…