Question 280259 in Docker Question posted in
The official documentation can be found here.

Docker – how to configure Qdrant data persistence and reload

I’m trying to build an app with streamlit that uses Qdrant python client.

to run the qdrant, im just using:

docker run -p 6333:6333 qdrant/qdrant

I have wrapped the client in something like this:

class Vector_DB:
    def __init__(self) -> None:
        self.collection_name = "__TEST__"
        self.client = QdrantClient("localhost", port=6333,path = "/home/Desktop/qdrant/qdrant.db")

but i’m getting this error:

Storage folder /home/Desktop/qdrant/qdrant.db is already accessed by
another instance of Qdrant client. If you require concurrent access,
use Qdrant server instead.

I suspect that streamlit is creating multiple instances of this class, but, if i try to load the db from one snapshot, like:

    class Vector_DB:
        def __init__(self) -> None:
             self.client = QdrantClient("localhost", port=6333)
             self.client.recover_snapshot(collection_name = "__TEST__",location = "http://localhost:6333/collections/__TEST__/snapshots/__TEST__-8742423504815750-2023-10-30-12-04-14.snapshot")

it works. Seems like i’m missing something important on how to configure it. What is the properly way of setting Qdrant, to store some embeddings, turn off the machine, and reload it?

Tags:

2

Answers


  1. 0

    When you specify the path value, the qdrant_client library provisions a local instance of Qdrant that doesn’t support concurrent access and is for testing only. The path refers to the path where the files of the local instance will be saved.

    In the second case, you didn’t specify the path value, which connects the client to the Docker instance that supports concurrent access and persistence as you’d expect.
    You can find the docs of the options here (It’s a work in progress though).
    https://python-client.qdrant.tech/qdrant_client.qdrant_client

    Login or Signup to reply.
  2. 0

    You mention using the Qdrant server, to which you’d like to connect with the Python client.

    There are two problems in your above question, let me go over both of them:

    1. Persist data in Qdrant server:
    A Qdrant server stores its data inside the Docker container. Docker containers are immutable however, which means that they don’t hold data across restarts. To persist data you must specify a mount. Qdrant will then persist data on the mount instead of in the immutable container. You could configure a mount using the -v flag like this1:

    docker run -p 6333:6333 
    -v $(pwd)/qdrant_storage:/qdrant/storage:z 
    qdrant/qdrant

    Data is automatically persisted and reloaded when you stop or restart the Qdrant container. You don’t have to take extra measures for this.

    2. Qdrant server versus local mode:
    Qdrant supports two operating modes. The Qdrant server and local mode. You’re using the Qdrant server through Docker. The Python client also supports local mode which is an in-memory implementation intended for testing.

    To use a Qdrant server you must specify its location (URL)2. You’ve already specified "localhost", perfect if hosting the Qdrant server on your local machine.

    To use local mode you can either specify ":memory:" or provide a path to persist data3.

    Right now you’ve specified parameters for both. Instead you must stick with one. You can update your client initialization to this:

    class Vector_DB:
    def __init__(self) -> None:
        self.collection_name = "__TEST__"
        self.client = QdrantClient("localhost", port=6333)
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search