Support Qdrant FineWeb-10B, PubMed-MV, and Coyo-VE datasets

Summary

Add the three vector-search datasets announced by Qdrant to VectorDBBench through the Hugging Face dataset-source and DatasetManager infrastructure introduced in #868:

• Qdrant/Coyo-VE
• Qdrant/PubMed-MV
• Qdrant/FineWeb-10B

They should use the existing public interface: a normal Performance case with the selected dataset name. A new case type should not be introduced.

Announcement: https://www.reddit.com/r/vectordatabase/comments/1w5j3zz/new_massive_and_diverse_vector_datasets_opened_to/

Dataset characteristics

Dataset	Corpus and vectors	Queries / ground truth	Initial integration assessment
Coyo-VE	15,380,795 vectors, 2,048-dimensional float32 dense embeddings, cosine	25,000 queries and exact top-1000 ground truth in one Parquet file	Best first target; dense vector search matches the current runner model
PubMed-MV	23,898,701 documents with 1,024-dimensional dense, sparse lexical, and token-level multivector representations; approximately 34.5 TB	1,000 queries with exact top-1000 ground truth per modality; multivector relevance uses MaxSim	Dense mode can be added first; sparse, multivector, hybrid, and reranking need new workload support
FineWeb-10B	10,074,324,060 documents with 768-dimensional dense plus sparse vectors; approximately 46.5 TB total	About 120,000 dense, sparse, and filtered queries with exact top-1000 ground truth	Requires query-vector preparation, filtering support, ID canonicalization, and an extreme-scale access strategy

Difficulties to resolve

1. Source IDs versus database IDs

All three datasets expose string document identifiers, while regular VectorDBBench vector clients commonly use integer primary keys. The integration needs a generic, deterministic ID translation mechanism shared by corpus insertion and ground-truth loading.

Requirements:

• Do not keep a corpus-wide mapping in memory for billion-scale datasets.
• Generate any mapping or sidecar before insertion timing starts.
• Keep mappings deterministic across runs and revisions.
• Coyo hit_ids join directly to corpus id.
• FineWeb corpus IDs use <urn:uuid:...>, while ground truth contains bare UUIDs, so canonicalization is required.
• Verify whether PubMed hit_ids can safely map through the corpus PMID column, including uniqueness and actual value format.

2. Parquet column projection and remote access

The current Hugging Face reader downloads complete matching files, and the Parquet iterator reads every column. This is especially problematic for PubMed because dense, sparse, and very large nested multivector columns coexist in the same shards.

The manager should read only the configured ID, vector, query, and ground-truth columns. For PubMed and FineWeb, investigate revision-pinned hf:///range-based Parquet reads or another resumable projected-access mechanism so a dense test does not download unrelated columns or an entire multi-terabyte repository.

Dataset preparation and downloads must remain outside measured insertion time.

3. Combined query and ground-truth artifacts

Coyo and PubMed store query vectors and ground truth in the same Parquet files. The role-based schema should allow one physical file to satisfy both query and ground-truth selectors without duplicating the download. Add fixtures covering this layout.

4. Vector modality and scoring support

The current vector runner assumes one dense vector per record. Full coverage requires explicit capability-gated workloads:

• Dense ANN search: suitable for the first implementation.
• Sparse vector search: needed for PubMed and FineWeb.
• Token-level multivector search with MaxSim: needed for PubMed.
• Hybrid/fusion and multistage reranking: needed to exercise PubMed's intended comparisons.

Unsupported database/workload combinations should fail during validation with a clear explanation rather than during insertion or search.

5. Filtered retrieval

FineWeb publishes both text/domain filters and structured filters over language score, dates, and crawl dumps. Registered Parquet datasets currently support only unfiltered performance cases. Supporting these queries requires a portable filter/payload schema, database-specific translation, filtered ground-truth selection, and capability validation.

6. FineWeb query vectors and licensing

FineWeb does not redistribute the MS MARCO query text or derived embeddings. Its scripts recover the queries and regenerate embeddings, but the dataset card notes that float32 regeneration can differ from the bfloat16 inputs used for the published ground truth and can affect results near the top-1000 boundary.

Before enabling FineWeb recall reporting, decide whether to:

• wait for an upstream reproducible query-vector artifact;
• regenerate vectors during dataset preparation and explicitly record the numerical limitation; or
• support corpus-only ingestion/index measurements until exact query compatibility is available.

The result metadata and documentation must also distinguish the ODC-BY corpus/ground-truth terms from the separate non-commercial MS MARCO query terms.

7. Scale-aware verification

Coyo is approximately 135 GB and is suitable for the first real end-to-end validation. Full PubMed and FineWeb downloads are not appropriate for regular CI or a single development smoke test.

Use layered verification:

1. Unit fixtures for selectors, schemas, combined query/GT files, ID translation, and provenance.
2. Small revision-pinned slices for download, insertion, search, and recall smoke tests.
3. A full Coyo dense run on an appropriately sized environment.
4. Manifest/schema/projected-read validation for PubMed and FineWeb before scheduling full-scale runs.

Proposed implementation sequence

Phase 1: generic dense Parquet support

• Add projected corpus/query/ground-truth column reads.
• Add deterministic, disk-backed source-ID translation and source-ID canonicalization hooks.
• Test a combined query-and-ground-truth Parquet file.
• Register Coyo-VE dense cosine under the existing Performance case.
• Record repository, immutable revision, vector modality, metric, model, dimensions, ID mapping, and license provenance in results.

Phase 2: PubMed dense

• Prove that dense-only reading does not fetch or materialize the nested multivector column.
• Validate PubMed corpus/ground-truth ID mapping.
• Register and smoke-test PubMed dense cosine search.

Phase 3: additional modalities and FineWeb

• Design capability-gated sparse-vector workloads.
• Design multivector/MaxSim and hybrid/multistage workloads.
• Add a portable filtered-retrieval schema for FineWeb.
• Resolve FineWeb query-vector reproducibility and licensing behavior.
• Define resumable, distributed-friendly full-scale execution for PubMed and FineWeb.

Acceptance criteria

• Each enabled dataset is selectable through the existing Performance case and dataset-name interface.
• Dataset revisions are immutable and recorded in exported results.
• Download/preparation time is excluded from insertion timing.
• Corpus vectors remain batch-streamed; queries and selected ground truth may be held in memory when bounded.
• Recall uses the published exact ground truth and rejects incompatible query vectors or ID mappings.
• Unsupported modalities, filters, or database clients fail during preflight validation with actionable messages.
• Tests cover schema failures, missing files, selector expansion, query/GT alignment, ID translation, and maximum supported k.