kubeflow / kubeflow/docs-agent

Bug: Pipeline drops entire Milvus collection on every run

Open
#10 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
42
Forks
111
Avg merge
6d 23m
Merged PRs (30d)
2

Description

## The Problem

In `pipelines/kubeflow-pipeline.py`:

```python
if utility.has_collection(collection_name):
utility.drop_collection(collection_name)
print(f"Dropped existing collection: {collection_name}")
```

Every pipeline run deletes ALL data and rebuilds from scratch. This means:
- You can't index multiple data sources (docs + issues)
- Re-running causes temporary data loss

## Why This Matters for Agentic RAG

The Agentic RAG GSoC proposal requires indexing multiple data sources:
- Kubeflow documentation
- GitHub Issues (for troubleshooting)
- Platform architecture docs

With the current code, running the issues pipeline would delete all docs. This blocks the proposal entirely. Fixing this is a prerequisite for building a multi-source RAG agent.

## Proposed Solutions

### Option 1: Source-Based Delete

Add a `source` field to every record, then delete only matching records before inserting.

```python
# Delete old data from this source only
collection.delete(expr=f'source == "docs:kubeflow/website"')
# Insert new data
collection.insert(records)
```

**Pros:** Simple concept.
**Cons:** Requires schema change, delete-by-expression scans all rows.

---

### Option 2: Milvus Partitions (Recommended)
Docs (https://milvus.io/docs/manage-partitions.md#Create-Partition)
Milvus has a first-class feature for this: **Partitions**. A partition is a subdivision of a collection that can be dropped/created independently.

```
Collection: docs_rag
├── Partition: docs__kubeflow_website (docs)
├── Partition: issues__kubeflow_kubeflow (issues)
└── Partition: issues__kubeflow_pipelines (issues)
```

Each pipeline run only drops and rebuilds **its own partition**:

```python
partition_name = f"{source_type}__{repo_name.replace('/', '_')}"

# Drop only this partition
if collection.has_partition(partition_name):
collection.drop_partition(partition_name)

# Create and populate
collection.create_partition(partition_name)
collection.insert(records, partition_name=partition_name)
```

**Pros:**
- No schema change needed
- O(1) drop (instant, no scanning)
- Milvus-native feature designed for this
- Search still works across all partitions automatically

```python
# Searches ALL partitions automatically
collection.search(query_vec, ...)
```

## Questions for Maintainers

1. Any concerns with switching to a partitioned collection?
2. Preference between Option 1 and Option 2?

I am happy to implement whichever approach makes sense.

Contributor guide

Open the contributing guide

Research direction

Start in pipelines/kubeflow-pipeline.py and inspect how the Milvus collection is created, dropped, and populated. Before implementing, resolve whether maintainers prefer source-based deletion or Milvus partitions. Done means rerunning one source preserves data from the other sources while refreshing only the selected source.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.