https-deeplearning-ai / https-deeplearning-ai/spec-build-lab
Next pass: candidate slate (A) + measurement watchlist (B)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3
- Forks
- 3
- Avg merge
- 20h 23m
- Merged PRs (30d)
- 6
Description
Forward-looking slate after PR #25. Two lists with different triggers: A needs a promotion pass (guide/spec edits + a validation round); B needs a tester round or a trial, not edits. Nothing here is applied.
Supersedes the forward halves of #18 and #24 (both closed). #19 (recording/compaction) stays separate — it is tooling, not spec/guide.
A. Candidates for the next promotion pass
Ordered by strength of evidence.
-
Harness-known identities are injected, never model-guessed. The agent-memory spec has this as an R11 sentence; the guide does not. Evidence it is needed: a clean-room regen against the current guide (regen-D) contains zero such language, so a spec generated today ships the same gap that made AC21 fail — the model, never told its thread id, guessed
"current_thread"and the summarize tool honestly consolidated nothing. Second datapoint: the first wording named one tool, and an independent trial build applied it to exactly that tool, leaving summary expansion unscoped. Write it concept-anchored (identities the harness knows are injected and absent from the model-facing schema; identifiers the model legitimately holds stay model-supplied). -
A shipped default that no acceptance criterion exercises (D10 augmentation). Tool-description augmentation is ON by default, but the offline stub cannot return augmentation JSON and no
liveAC registers through a real model — so 2 of 2 builds shipped it unexercised. Compare R5, which the spec deliberately made stub-testable. Candidate rule: a Ledger default that changes behavior needs at least one AC that exercises that default, offline-testable or explicitlylive. -
An AC depending on a fixture the spec never declares. AC7 and AC11 require a "scripted summarizer stub"; §5 declares four stubs and that is not one of them — in a section that opens "MUST create these fixtures exactly as described." 2 of 2 builds invented it, so part of the offline oracle rests on builder-authored fixture content. Candidate: §14 audit line — every fixture an AC names is declared in §5.
-
Contract constraints with no AC behind them.
SummaryRecord.descriptionminLength 8 and R6's 8–12-word label band are stated in prose/schema and never asserted. Candidate: the existing rules↔ACs bidirectional audit extends to schema constraints that encode behavior. -
Prose-economy sweep. This batch grew the guide 11,041 → 12,313 words (+11.5%) and the spec 9,773 → 10,528 (+7.7%), concentrated in §0/§6.0 — the sections three testers called too dense. At this rate the guide doubles roughly every six batches. Candidate: a pass whose goal is subtraction (merge overlapping rules, move rationale out of emitted templates, delete restated mechanics) rather than new rules, with the regen properties as the safety net.
-
F8 — fixture-corpus churn across regenerations. Parked by owner decision, carried here so it is not lost. Each regeneration authors fresh synthetic fixtures by design, so fixture identities churn and cross-generation comparisons carry noise. Not currently biting: with the Option-B pattern (surgical spec edits, no regen adoption) regenerations are evidence, not products.
B. Measurement watchlist
Needs a tester round or a disposable-clone trial. No edits pending.
- Does the orientation block cure the density complaint? Three testers found the spec overwhelming up front (R16: "a brief, high-level overview at the top"; R18: "I needed a better explanations and instructions in the beginning"). The fix shipped in #25 and is unproven with humans — a build agent reusing the prose is not evidence.
- Does the one-question gate feel light enough? R17 objected to "the heavy question-tool ceremony", and F12 replaced 13 questions with one for the default path. Needs a human reaction, not a probe.
- Baseline resource benchmark (wall-clock, tokens, disk) for a recommended-baseline build. This is the answer to the cost-abandonment feedback, which the owner decided to measure rather than promise in spec text.
- Does a build actually write
resolved-decisions.md? Accepted residual from #25: the requirement is validated in text, the behavior is not. Rides the next trial rather than a paid probe build. - Ingestion idempotency — one tester hit UNIQUE errors re-running setup. The current build cannot crash that way (generated ids) but duplicates silently. Proposal recorded, unapplied: an idempotent-seed AC or an explicit "re-seeding duplicates; reset first" note.
- arXiv 429 recurrence — transient rate limits hit a tester and our own runs; the client retries internally and R11 makes a mid-loop 429 non-fatal. Proposal recorded, unapplied: document transient-429 as expected for
live (net)criteria rather than mandate backoff code.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Treat this as a planning and measurement watchlist rather than a single starter task. Read PR #25, the agent-memory spec and guide, then compare regen-D, AC7/AC11, D10, and the §5/§14 audits. A follow-up is ready when one candidate has a bounded edit and validation plan; watchlist items require a tester round or disposable-clone trial.
Written by the indexing model from the issue text.
Assessment
- Domain
- documentation, testing
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100