Pathological single-statement input can hang indexing for tens of minutes — tree-sitter progress callback doesn't fire during error recovery
- Dominant language
- Rust
- Stars
- 88
- Forks
- 16
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
A single large source file — specifically one whose grammar produces a very small number of statements relative to file size (e.g. a SQL file with only a handful of top-level statements but each spanning megabytes, such as one `INSERT ... VALUES (...), (...), ...` with tens of thousands of value-tuples) — can cause `infigraph index` to hang for tens of minutes on that one file, without finishing.
Live profiling (macOS `sample`, repeated across multiple runs) shows the parsing thread stuck almost entirely inside tree-sitter's internal error-recovery path:
```
extract::extract_file
→ ts_parser_parse_with_options
→ ts_parser_parse
→ ts_parser__recover
→ ts_subtree_summarize_children
→ stack_node_release / ts_subtree_release
```
This matches tree-sitter's GLR error-recovery machinery thrashing on a single large, ambiguous statement rather than making forward progress.
## Root cause: progress callback does not reliably fire during recovery
`infigraph-core`'s `extract_file` (`crates/infigraph-core/src/extract/mod.rs`) uses tree-sitter's `parse_with_options` with a `progress_callback` (the mechanism tree-sitter 0.25+ recommends for parse cancellation, replacing the older `set_timeout_micros`, which was removed entirely by 0.26). The callback tracks `ParseState::current_byte_offset()` and requests cancellation (`ControlFlow::Break`) once byte-offset progress has stalled for a grace window.
This was implemented and tested against a real reproduction of the hang:
- On one run, the callback correctly detected the stall and cancelled the parse within the expected window.
- On repeated identical runs against the exact same input and binary, the callback did **not** fire at all — profiling during the hang showed zero samples anywhere near the callback/checkpoint, only deep inside `ts_parser__recover`'s reduction/release internals, for the entire duration (multiple minutes, would otherwise have continued far longer based on the original observed ~40 minute hang).
This indicates tree-sitter's error-recovery loop does not consistently return to the checkpoint where the progress callback is invoked — the callback-based cancellation mechanism appears to be unreliable specifically during recovery, as opposed to normal parsing progress. Cancellation therefore cannot be counted on to bound worst-case parse time when a file triggers recovery-heavy behavior.
## What's needed
A real fix would require either:
- A tree-sitter-level change so recovery genuinely checks/respects the progress callback (or an equivalent cancellation point) on a bounded cadence regardless of how deep or how long the recovery loop runs, or
- Confirmation/guidance from tree-sitter's side on whether there's a supported way to bound recovery-loop duration that we're missing.
We deliberately have not shipped a workaround (e.g. skipping oversized files) as the fix for this — we'd like to understand whether a real, dependency-level fix is feasible before falling back to a heuristic in our own extraction pipeline.
## Environment
- tree-sitter crate: 0.26 (workspace pins `tree-sitter = "0.26"`; confirmed same behavior against 0.26.11, the current latest release)
- Platform observed: macOS (arm64)
- Rust bindings: `parse_with_options` / `ParseOptions::progress_callback` / `ParseState::current_byte_offset`
Contributor guide
Research direction
Start in crates/infigraph-core/src/extract/mod.rs and inspect the tree-sitter 0.26 parse_with_options progress_callback using ParseState::current_byte_offset. Reproduce the large single-statement input and compare runs where the callback fires with runs stuck in recovery. Done means establishing a supported dependency-level way to bound recovery, or documenting tree-sitter guidance before considering a workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- compilers, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100