[dotnet-ai] Vector data search + data ingestion pipeline
- Dominant language
- C#
- Stars
- 5.4k
- Forks
- 415
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 81
Description
## Parent Issue
Part of the `dotnet-ai` plugin proposal: #225
## Summary
Add the `vector-data-search` and `data-ingestion-pipeline` skills to the `dotnet-ai` plugin.
These two skills are tightly related — data ingestion produces chunks that get embedded and stored in vectors, which vector search then queries. They form the data infrastructure layer of the RAG stack and are submitted together as one deliverable.
## Skills in This Issue
### vector-data-search
Guides developers through using `Microsoft.Extensions.VectorData` abstractions for storing, indexing, and querying vector embeddings.
**Covers:**
- `IVectorStore` and `IVectorStoreRecordCollection` abstractions
- Connector selection (Azure AI Search, Qdrant, Milvus, Weaviate, in-memory for dev/test)
- Record model definition with `[VectorStoreRecordKey]`, `[VectorStoreRecordData]`, `[VectorStoreRecordVector]` attributes
- Collection creation, upsert, vector search with `VectorSearchAsync()`
- Hybrid search (vector + keyword) where connectors support it
- Provenance metadata — storing source document info for attribution in RAG
**Does NOT cover:**
- Generating embeddings (use `meai-embeddings`)
- Document ingestion/chunking (use `data-ingestion-pipeline`)
- End-to-end RAG (use `rag-pipeline`)
### data-ingestion-pipeline
Guides developers through building document ingestion pipelines using `Microsoft.Extensions.DataIngestion`.
**Covers:**
- Pipeline builder pattern — `DataIngestionPipelineBuilder`
- Document loading from files, URIs, and custom sources
- Chunking strategies — fixed-size, sentence-based, paragraph-based, recursive
- Chunk enrichment — adding metadata, summaries, or computed fields
- Error handling — partial failures, retry on transient errors, dead-letter handling
- Integration with embeddings (IEmbeddingGenerator) and vector storage (IVectorStore)
**Does NOT cover:**
- Tabular/structured data processing (use `mlnet`)
- Chat or LLM interactions (use `meai-chat-integration`)
- End-to-end RAG (use `rag-pipeline`)
## Files
```
plugins/dotnet-ai/skills/vector-data-search/SKILL.md
plugins/dotnet-ai/skills/data-ingestion-pipeline/SKILL.md
tests/dotnet-ai/vector-data-search/eval.yaml
tests/dotnet-ai/data-ingestion-pipeline/eval.yaml
```
## Eval Scenarios
**vector-data-search:**
1. **Set up vector search** — prompt asks to store and search product embeddings; expects VectorData abstractions with connector selection and record model
**data-ingestion-pipeline:**
1. **Build document ingestion pipeline** — prompt asks to ingest PDF documents for RAG; expects DataIngestion pipeline with chunking, enrichment, and integration with vector store
## Key Packages
- `Microsoft.Extensions.VectorData` (abstractions)
- Connector: `Microsoft.Extensions.VectorData.AzureAISearch`, `Microsoft.Extensions.VectorData.Qdrant`, etc.
- `Microsoft.Extensions.DataIngestion`
## Dependencies
Requires the plugin scaffold (sub-issue #1) to be merged first.
## Acceptance Criteria
- [ ] Both SKILL.md files have valid YAML frontmatter
- [ ] vector-data-search covers IVectorStore, record model, connector selection, hybrid search
- [ ] data-ingestion-pipeline covers pipeline builder, chunking strategies, enrichment, error handling
- [ ] Both eval.yaml files have at least one scenario
- [ ] Cross-references between the two skills are correct
Contributor guide
Research direction
Start with the plugin scaffold dependency and inspect the two target paths: plugins/dotnet-ai/skills/vector-data-search/SKILL.md and plugins/dotnet-ai/skills/data-ingestion-pipeline/SKILL.md. Review the listed Microsoft.Extensions.VectorData and Microsoft.Extensions.DataIngestion packages, then create both eval.yaml files from their scenarios. Done means both skills have valid frontmatter, cover the stated topics, include at least one eval scenario, and cross-reference each other correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100