norrisaftcc / norrisaftcc/the_algorithm
Patch-up run: re-fire only the unrun cells, to split instrument from model
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Explicitly out of scope. Part of #12. For a later session, after PR #9 lands. Posted to the winds — not to be run now. Frozen from the Algorithm's seat (signer: TEACHERBOT-BLUE).
Re-fire only the unrun cells, to split instrument failure from model failure
Many fails in this battery are not the model. A cell that never emits a
visible reply — truncated, errored, or precondition-unmet — tells you about
the venue and the budget, not the doctrine. The cheapest way to learn which
is to re-run those cells alone, with the two known venue defects fixed, and
compare. A pass that appears only after the ceiling rises was always an
instrument failure; a fail that survives is the model's.
Keep it small. Do not re-run the whole matrix. Select cells by their
recorded outcome class, fix the ceiling and the retry, pin the provider,
and set a low budget that aborts.
- Scope: re-run only cells with outcome truncated, error, or n/a-precondition. Skip every scored pass and fail.
- Reasoning starvation: raise max_tokens per model. Reasoning tokens bill against the ceiling; at 2000 the reply starves. gpt-5-mini lost most of its rows this way.
- Rate limit: retry HTTP 429 with backoff. qwen P18 r2 errored on a 429.
- Provider drift: pin one provider per model. qwen ran on two providers under allow_fallbacks:false.
- Budget: only the unrun cells fire, so cost is small. Set a low ceiling and abort on it.
- Out of scope now: for a later session, after PR #9 lands.
Open questions
- Can the harness select cells by recorded outcome class for a re-run?
- What max_tokens ceiling for reasoning models — per model, or one raised global?
- What retry count and backoff for a 429?
- How is single-provider enforced when allow_fallbacks:false did not hold?
- Should a later skill version carry a venue control — a pre-check that the model can emit a visible reply under the ceiling, before any probe verdict is read? Then a starved reply is separated from a refusal by construction, not by inspection.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing issue #12 and wait for PR #9 to land, as this work is explicitly deferred. Define how the harness selects truncated, error, and n/a-precondition outcomes, then resolve the questions about reasoning-token ceilings, 429 backoff, provider pinning, and budget aborts. Done means only those cells are rerun with the venue defects fixed and scored passes and fails are skipped.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100