kubeflow / kubeflow/docs-agent

Issues pipeline chunk_size (1500) exceeds the TEI embedding limit (1000) — tail of long chunks is unsearchable

Open Beginner friendly
#216 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
42
Forks
111
Avg merge
6d 23m
Merged PRs (30d)
2

Description

### Problem

The GitHub issues ingestion pipeline chunks issue content up to **1500 characters**, but truncates every chunk to **1000 characters** before sending it to the TEI embeddings service:

- `docs-agent-mcp/pipelines/issues-pipeline.py#L481` — `chunk_size: int = 1500`
- `docs-agent-mcp/pipelines/issues-pipeline.py#L301` — `max_tei_chars = 1000`, applied as `r["content_text"][:max_tei_chars]`

So for any chunk longer than 1000 chars, the **last ~500 characters are stored in Milvus and shown to users, but were never embedded** — that text is invisible to semantic search.

Contributor guide

Open the contributing guide

Research direction

Start in docs-agent-mcp/pipelines/issues-pipeline.py at lines 301 and 481, tracing how chunk_size and max_tei_chars flow through ingestion. Verify that the resulting chunk content and TEI input cover the same text, then run the pipeline's available validation or ingestion checks to confirm long issue chunks are fully searchable.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, search
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.