kubeflow / kubeflow/docs-agent
Issues pipeline chunk_size (1500) exceeds the TEI embedding limit (1000) — tail of long chunks is unsearchable
- Dominant language
- Python
- Stars
- 42
- Forks
- 111
- Avg merge
- 6d 23m
- Merged PRs (30d)
- 2
Description
### Problem
The GitHub issues ingestion pipeline chunks issue content up to **1500 characters**, but truncates every chunk to **1000 characters** before sending it to the TEI embeddings service:
- `docs-agent-mcp/pipelines/issues-pipeline.py#L481` — `chunk_size: int = 1500`
- `docs-agent-mcp/pipelines/issues-pipeline.py#L301` — `max_tei_chars = 1000`, applied as `r["content_text"][:max_tei_chars]`
So for any chunk longer than 1000 chars, the **last ~500 characters are stored in Milvus and shown to users, but were never embedded** — that text is invisible to semantic search.
Contributor guide
Research direction
Start in docs-agent-mcp/pipelines/issues-pipeline.py at lines 301 and 481, tracing how chunk_size and max_tei_chars flow through ingestion. Verify that the resulting chunk content and TEI input cover the same text, then run the pipeline's available validation or ingestion checks to confirm long issue chunks are fully searchable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, search
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100