Truncation messages bypass the warning system and repeat per chunk
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 17
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
When a chunk's token count exceeds the embedder's cap, encode in src/engine/parallel_embedder.rs prints a line and moves on:
if token_count > MAX_LENGTH {
eprintln!(
"Warning: embedding input truncated from {} to {} tokens ({} discarded)",
token_count,
MAX_LENGTH,
token_count - MAX_LENGTH
);
}
It is the only warning in the crawl that is not a CrawlWarning, which costs three things. It prints once per chunk from parallel workers, so a file that produces forty oversized chunks produces forty lines interleaved with everything else. It names no file, so a reader cannot tell which file to look at. And it is invisible to the crawl summary, so a crawl that truncated ten thousand chunks reports the same summary as one that truncated none.
@davidh233's team ran into the practical version of this on a 39-hour crawl of a 230k-file monorepo: the messages arrive from parallel workers across the whole run, and the handful of cases anyone would want to investigate are buried in thousands of repetitions of cases nobody cares about.
A chunk that hits the cap is usually minified output, generated code, or a file with very long single lines, and the fix is normally a patternsToExclude entry rather than anything in the code. That decision requires knowing which files, and today the output does not say.
Proposal
Add a variant to CrawlWarning in src/engine/warning.rs carrying the file path and the count, emitted once per file rather than once per chunk:
/// Embedding input exceeded the token cap and was truncated.
EmbeddingInputTruncated {
relative_path: String,
chunk_count: usize,
max_tokens: usize,
},
That puts it under the same rendering and counting as the existing four variants, and the crawl summary gains a line naming the affected files. The stored text is unaffected either way, so FTS still covers the truncated tail; what is lost is only the vector's coverage of it.
Two implementation notes. encode takes &self and worker_index and returns a vector, with no warning sink in scope, so the truncation has to be reported back to the caller in src/app/crawl/pipeline.rs rather than emitted where it is detected. And per-file aggregation means the count accumulates across workers before anything is emitted, since chunks of one file are spread across the pool.
Not in this issue
Whether truncation warnings can be muted. Chunk-level warnings like chunker-fallback are mutable because a stored column identifies which chunks are affected, so a mute verb can ask the database which paths are currently degraded. Truncation has no such marker: the embedder truncates and moves on, and nothing about it is written down. Making it mutable would need either a column recorded at chunk-write time or a second way for the mute verb to derive its candidates, and that belongs with the mute design rather than here.
Whether these failures should affect crawl_complete. Truncation is a degradation rather than a failure, so it does not, but the broader question of which per-file problems should prevent a crawl from reporting success is open and separate.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with src/engine/warning.rs and the encode entry point in src/engine/parallel_embedder.rs, then trace reporting through src/app/crawl/pipeline.rs. Verify that oversized chunks are aggregated per file across workers, rendered and counted with the existing CrawlWarning variants, and included in the crawl summary without changing stored text or crawl completion status.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, devtools
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100