Static TASK_PATTERNS preempt a stronger keyword-overlap match, costing ~4pp routing accuracy
- Dominant language
- TypeScript
- Stars
- 72.7k
- Forks
- 8.6k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 83
Description
`hooks route` picks the first semantic match clearing `score > 0.4`. Static
patterns need only that bar; learned patterns additionally need
`support >= 2 && reliability >= 0.75` (hooks-tools.js, the #2864 comment).
The problem is not the learned gate — it is that a static pattern winning on
score **skips `suggestAgentsForTask` entirely**, and that function contains a
second, much stronger learned stage: nearest-neighbour over
`routing-outcomes.json` by keyword overlap (>= 2 required). A static pattern
matching on a weak char-hash similarity therefore beats an outcome match with
14 overlapping keywords.
**Measured** on a 75-outcome memorization replay (ruflo 3.33.0, both backends):
| decided by | accuracy | n |
|---|---|---|
| keyword-fallback | 53% | 57 |
| learned-* | 67% | 9 |
| **static TASK_PATTERNS** | **33%** | **9** |
Static is the least accurate path and wins 9 routes. Simulating
`suggestAgentsForTask` on those exact 9 tasks (same KEYWORD_PATTERNS table,
same stopword tokenizer, same >=2 overlap rule):
```
static today : 3/9 correct
if fallback : 6/9 correct net +3 routes
overall : 39/75 (52%) -> 42/75 (56%)
```
The three flips, all currently lost to a static pattern:
```
want tester static gave security-architect fallback: tester (KEYWORD:test)
want tester static gave architect fallback: tester (OUTCOME-OVERLAP:7)
want coder static gave architect fallback: coder (OUTCOME-OVERLAP:14)
```
An outcome match with 14 shared keywords losing to `refactor-task` is the
clearest case.
**Suggested fix:** require a static pattern to beat the best
`suggestAgentsForTask` result rather than short-circuiting it — or gate static
patterns on a higher score bar than learned ones, since they carry no
support/reliability evidence. Either restores the ordering the #2864 fix was
reaching for.
**Secondary:** `buildLearnedRoutingPatterns` accepts `LearnedRoutingOptions`,
but the only caller (`loadLearnedPatterns`) passes none and no env/config
override exists, so `minKeywordSupport`, `minDiscriminativeShare` and the `0.4`
bar cannot be tuned without a code change.
Environment: ruflo v3.33.0, macOS, 75 labelled outcomes. Identical results on
native HNSW and `CLAUDE_FLOW_DISABLE_NATIVE_ROUTER=1`.
Contributor guide
Research direction
Start in hooks-tools.js by tracing the static TASK_PATTERNS path and suggestAgentsForTask, then inspect routing-outcomes.json and the loadLearnedPatterns/buildLearnedRoutingPatterns call chain. Reproduce the 75-outcome comparison described in the issue and verify that static matches no longer bypass the stronger learned result, with routing accuracy improving from 52% toward 56%.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ai
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100