qdrant / qdrant/qdrant-client

High Disk Usage for Simple PDF with Qdrant Collection

Open
#867 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
304
Avg merge
2d 9h
Merged PRs (30d)
37

Description

While using Qdrant for managing document embeddings, I observed an issue with disk space utilization. When processing a very simple PDF file, the created collection occupies significantly high disk space, even though the snapshot size remains small.

Code Snippet
Below is the code used for creating and loading the collection:

Python

from langchain_qdrant import QdrantVectorStore
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_huggingface import HuggingFaceEmbeddings
from pymupdf_loader import PyMuPDFLoader
from qdrant_client.models import VectorParams, Distance
from qdrant_client import QdrantClient, models

collection_name = "xyx"
embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L12-v2")
embedding_size = len(embeddings.embed_query("Sample text"))
upload_path = "path of pdf"
loader = PyMuPDFLoader(upload_path)

docs = loader.load()
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
splits = text_splitter.split_documents(docs)

qdrant_client.create_collection(
    collection_name=collection_name,
    vectors_config=VectorParams(size=embedding_size, distance=Distance.COSINE, on_disk=True),
)

vector_store = QdrantVectorStore(
    client=qdrant_client,
    collection_name=collection_name,
    embedding=embeddings,
)
vector_store.add_documents(documents=splits)

Issue
After processing and loading vectors for a very simple PDF document:

The on-disk size of the collection is unexpectedly high.
The snapshot size, however, is relatively small.

Observed Behavior
The discrepancy between the on-disk size and the snapshot size indicates potential inefficiency in storage utilization or metadata overhead.

This behavior might impact scenarios where multiple small PDFs are processed, leading to disproportionately high disk usage.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported behavior with the Python snippet, focusing on create_collection with on_disk=True and the subsequent add_documents call. Compare the collection's on-disk size with its snapshot size for the minimal PDF, then trace the client request path to determine whether the discrepancy originates in the client configuration or Qdrant storage; document the measurements and likely cause.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.