Question 403088 in Ubuntu Question posted in
The official Ubuntu documentation can be found here.

Ubuntu – How to use ChatOpenAI with LM Studio for LLMs? (Langchain)

I want to use ChatOpenAI from langchain-openai package to communicate with my local LM Studio server on Ubuntu 22.

I followed docs and came up with this – which is a standard way:

from langchain_openai import ChatOpenAI
from dotenv import load_dotenv

load_dotenv()

llm = ChatOpenAI(
    api_key=os.getenv("OPENAI_API_KEY"),
    base_url=os.getenv("OPENAI_API_BASE_URL"),
    temperature=0.5,
)

where envs are:

OPENAI_API_BASE_URL="http://localhost:1234/v1"
OPENAI_API_KEY="lm-studio"

But when I actually use that llm I get an error:

Traceback (most recent call last):
  File "<PROJECT_PATH>/web_researcher/main.py", line 155, in <module>
    result = crew.run()
             ^^^^^^^^^^
  File "<PROJECT_PATH>/web_researcher/main.py", line 114, in run
    result = crew.kickoff()
             ^^^^^^^^^^^^^^
  File "<COMPUTER_PATH>/.cache/pypoetry/virtualenvs/project-name-977PB1Et-py3.11/lib/python3.11/site-packages/langtrace_python_sdk/instrumentation/crewai/patch.py", line 153, in traced_method
    result = wrapped(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "<COMPUTER_PATH>/.cache/pypoetry/virtualenvs/project-name-977PB1Et-py3.11/lib/python3.11/site-packages/crewai/crew.py", line 469, in kickoff
    result = self._run_sequential_process()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<COMPUTER_PATH>/.cache/pypoetry/virtualenvs/project-name-977PB1Et-py3.11/lib/python3.11/site-packages/crewai/crew.py", line 577, in _run_sequential_process
    return self._execute_tasks(self.tasks)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<COMPUTER_PATH>/.cache/pypoetry/virtualenvs/project-name-977PB1Et-py3.11/lib/python3.11/site-packages/crewai/crew.py", line 665, in _execute_tasks
    task_output = task.execute_sync(
                  ^^^^^^^^^^^^^^^^^^
  File "<COMPUTER_PATH>/.cache/pypoetry/virtualenvs/project-name-977PB1Et-py3.11/lib/python3.11/site-packages/crewai/task.py", line 180, in execute_sync
    return self._execute_core(agent, context, tools)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<COMPUTER_PATH>/.cache/pypoetry/virtualenvs/project-name-977PB1Et-py3.11/lib/python3.11/site-packages/crewai/task.py", line 234, in _execute_core
    result = agent.execute_task(
             ^^^^^^^^^^^^^^^^^^^
  File "<COMPUTER_PATH>/.cache/pypoetry/virtualenvs/project-name-977PB1Et-py3.11/lib/python3.11/site-packages/langtrace_python_sdk/instrumentation/crewai/patch.py", line 153, in traced_method
    result = wrapped(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "<COMPUTER_PATH>/.cache/pypoetry/virtualenvs/project-name-977PB1Et-py3.11/lib/python3.11/site-packages/crewai/agent.py", line 182, in execute_task
    memory = contextual_memory.build_context_for_task(task, context)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<COMPUTER_PATH>/.cache/pypoetry/virtualenvs/project-name-977PB1Et-py3.11/lib/python3.11/site-packages/crewai/memory/contextual/contextual_memory.py", line 24, in build_context_for_task
    context.append(self._fetch_stm_context(query))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<COMPUTER_PATH>/.cache/pypoetry/virtualenvs/project-name-977PB1Et-py3.11/lib/python3.11/site-packages/crewai/memory/contextual/contextual_memory.py", line 33, in _fetch_stm_context
    stm_results = self.stm.search(query)
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "<COMPUTER_PATH>/.cache/pypoetry/virtualenvs/project-name-977PB1Et-py3.11/lib/python3.11/site-packages/crewai/memory/short_term/short_term_memory.py", line 33, in search
    return self.storage.search(query=query, score_threshold=score_threshold)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<COMPUTER_PATH>/.cache/pypoetry/virtualenvs/project-name-977PB1Et-py3.11/lib/python3.11/site-packages/crewai/memory/storage/rag_storage.py", line 102, in search
    else self.app.search(query, limit)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<COMPUTER_PATH>/.cache/pypoetry/virtualenvs/project-name-977PB1Et-py3.11/lib/python3.11/site-packages/embedchain/embedchain.py", line 699, in search
    return [{"context": c[0], "metadata": c[1]} for c in self.db.query(**params)]
                                                         ^^^^^^^^^^^^^^^^^^^^^^^
  File "<COMPUTER_PATH>/.cache/pypoetry/virtualenvs/project-name-977PB1Et-py3.11/lib/python3.11/site-packages/embedchain/vectordb/chroma.py", line 220, in query
    result = self.collection.query(
             ^^^^^^^^^^^^^^^^^^^^^^
  File "<COMPUTER_PATH>/.cache/pypoetry/virtualenvs/project-name-977PB1Et-py3.11/lib/python3.11/site-packages/langtrace_python_sdk/instrumentation/chroma/patch.py", line 91, in traced_method
    result = wrapped(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "<COMPUTER_PATH>/.cache/pypoetry/virtualenvs/project-name-977PB1Et-py3.11/lib/python3.11/site-packages/chromadb/api/models/Collection.py", line 327, in query
    valid_query_embeddings = self._embed(input=valid_query_texts)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<COMPUTER_PATH>/.cache/pypoetry/virtualenvs/project-name-977PB1Et-py3.11/lib/python3.11/site-packages/chromadb/api/models/Collection.py", line 633, in _embed
    return self._embedding_function(input=input)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<COMPUTER_PATH>/.cache/pypoetry/virtualenvs/project-name-977PB1Et-py3.11/lib/python3.11/site-packages/chromadb/api/types.py", line 193, in __call__
    result = call(self, input)
             ^^^^^^^^^^^^^^^^^
  File "<COMPUTER_PATH>/.cache/pypoetry/virtualenvs/project-name-977PB1Et-py3.11/lib/python3.11/site-packages/chromadb/utils/embedding_functions.py", line 188, in __call__
    embeddings = self._client.create(
                 ^^^^^^^^^^^^^^^^^^^^
  File "<COMPUTER_PATH>/.cache/pypoetry/virtualenvs/project-name-977PB1Et-py3.11/lib/python3.11/site-packages/langtrace_python_sdk/instrumentation/openai/patch.py", line 467, in traced_method
    result = wrapped(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "<COMPUTER_PATH>/.cache/pypoetry/virtualenvs/project-name-977PB1Et-py3.11/lib/python3.11/site-packages/openai/resources/embeddings.py", line 114, in create
    return self._post(
           ^^^^^^^^^^^
  File "<COMPUTER_PATH>/.cache/pypoetry/virtualenvs/project-name-977PB1Et-py3.11/lib/python3.11/site-packages/openai/_base_client.py", line 1271, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<COMPUTER_PATH>/.cache/pypoetry/virtualenvs/project-name-977PB1Et-py3.11/lib/python3.11/site-packages/openai/_base_client.py", line 942, in request
    return self._request(
           ^^^^^^^^^^^^^^
  File "<COMPUTER_PATH>/.cache/pypoetry/virtualenvs/project-name-977PB1Et-py3.11/lib/python3.11/site-packages/openai/_base_client.py", line 1046, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.AuthenticationError: Error code: 401 - {'error': {'message': 'Incorrect API key provided: lm-studio. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}

System Info

Ubuntu 22

Versions:

  1. Python: 3.11.5

  2. packages:

  • openai – 1.39.0
  • langchain – 0.2.12
  • langchain-cohere – 0.1.9
  • langchain-community – 0.2.11
  • langchain-core – 0.2.28
  • langchain-exa – 0.1.0
  • langchain-experimental – 0.0.64
  • langchain-openai – 0.1.20
  • langchain-text-splitters – 0.2.2

Perhaps I’m doing something wrong but I don’t think so, I read many docs and web posts and all show this is the correct way.

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    I figured out what the problem is and I was able to fix it.

    I use LM Studio as a server for LLMs. When my agents have the param memory set to true, agents use RAG to manage long and short term memory to improve the performance of the crew.

    By default, the crew is set to use OpenAI embeddings therefore it tries to connect to OpenAI even though LM Studio is used for LLMs.

    Unfortunately, CrewAI isn't yet adjusted to use LM Studio for embeddings therefore for local embeddings we can only use Ollama.

    Then we need to analyze our crew:

            crew = Crew(
            agents=[...],
            tasks=[...],
            process=Process.sequential,
            manager_llm=self.LM_Studio_Llama31_8B_LLM,
            embedder={
                "provider": "ollama",
                "config": {
                    "model": "mxbai-embed-large",
                }
            },
            output_log_file="final_logs.txt",
            share_crew=False,
            verbose=True,
        )

    where mxbai-embed-large is the embedding model that runs on the Ollama server.

    Now, no request will go to OpenAI.


  2. 0

    it looks like there is some issue with your openai API key

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search