dotnet / dotnet/skills

[dotnet-ai] Vector data search + data ingestion pipeline

Open
#230 0 comments 0 reactions 0 assignees View on GitHub
area-ai skill
Dominant language
C#
Stars
5.4k
Forks
415
Avg merge
1d 5h
Merged PRs (30d)
81

Description

## Parent Issue

Part of the `dotnet-ai` plugin proposal: #225

## Summary

Add the `vector-data-search` and `data-ingestion-pipeline` skills to the `dotnet-ai` plugin.

These two skills are tightly related — data ingestion produces chunks that get embedded and stored in vectors, which vector search then queries. They form the data infrastructure layer of the RAG stack and are submitted together as one deliverable.

## Skills in This Issue

### vector-data-search

Guides developers through using `Microsoft.Extensions.VectorData` abstractions for storing, indexing, and querying vector embeddings.

**Covers:**
- `IVectorStore` and `IVectorStoreRecordCollection` abstractions
- Connector selection (Azure AI Search, Qdrant, Milvus, Weaviate, in-memory for dev/test)
- Record model definition with `[VectorStoreRecordKey]`, `[VectorStoreRecordData]`, `[VectorStoreRecordVector]` attributes
- Collection creation, upsert, vector search with `VectorSearchAsync()`
- Hybrid search (vector + keyword) where connectors support it
- Provenance metadata — storing source document info for attribution in RAG

**Does NOT cover:**
- Generating embeddings (use `meai-embeddings`)
- Document ingestion/chunking (use `data-ingestion-pipeline`)
- End-to-end RAG (use `rag-pipeline`)

### data-ingestion-pipeline

Guides developers through building document ingestion pipelines using `Microsoft.Extensions.DataIngestion`.

**Covers:**
- Pipeline builder pattern — `DataIngestionPipelineBuilder`
- Document loading from files, URIs, and custom sources
- Chunking strategies — fixed-size, sentence-based, paragraph-based, recursive
- Chunk enrichment — adding metadata, summaries, or computed fields
- Error handling — partial failures, retry on transient errors, dead-letter handling
- Integration with embeddings (IEmbeddingGenerator) and vector storage (IVectorStore)

**Does NOT cover:**
- Tabular/structured data processing (use `mlnet`)
- Chat or LLM interactions (use `meai-chat-integration`)
- End-to-end RAG (use `rag-pipeline`)

## Files

```
plugins/dotnet-ai/skills/vector-data-search/SKILL.md
plugins/dotnet-ai/skills/data-ingestion-pipeline/SKILL.md
tests/dotnet-ai/vector-data-search/eval.yaml
tests/dotnet-ai/data-ingestion-pipeline/eval.yaml
```

## Eval Scenarios

**vector-data-search:**
1. **Set up vector search** — prompt asks to store and search product embeddings; expects VectorData abstractions with connector selection and record model

**data-ingestion-pipeline:**
1. **Build document ingestion pipeline** — prompt asks to ingest PDF documents for RAG; expects DataIngestion pipeline with chunking, enrichment, and integration with vector store

## Key Packages

- `Microsoft.Extensions.VectorData` (abstractions)
- Connector: `Microsoft.Extensions.VectorData.AzureAISearch`, `Microsoft.Extensions.VectorData.Qdrant`, etc.
- `Microsoft.Extensions.DataIngestion`

## Dependencies

Requires the plugin scaffold (sub-issue #1) to be merged first.

## Acceptance Criteria

- [ ] Both SKILL.md files have valid YAML frontmatter
- [ ] vector-data-search covers IVectorStore, record model, connector selection, hybrid search
- [ ] data-ingestion-pipeline covers pipeline builder, chunking strategies, enrichment, error handling
- [ ] Both eval.yaml files have at least one scenario
- [ ] Cross-references between the two skills are correct

Contributor guide

Open the contributing guide

Research direction

Start with the plugin scaffold dependency and inspect the two target paths: plugins/dotnet-ai/skills/vector-data-search/SKILL.md and plugins/dotnet-ai/skills/data-ingestion-pipeline/SKILL.md. Review the listed Microsoft.Extensions.VectorData and Microsoft.Extensions.DataIngestion packages, then create both eval.yaml files from their scenarios. Done means both skills have valid frontmatter, cover the stated topics, include at least one eval scenario, and cross-reference each other correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.