https-deeplearning-ai / https-deeplearning-ai/spec-build-lab

Next pass: candidate slate (A) + measurement watchlist (B)

Open
#26 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
3
Forks
3
Avg merge
20h 23m
Merged PRs (30d)
6

Description

Forward-looking slate after PR #25. Two lists with different triggers: A needs a promotion pass (guide/spec edits + a validation round); B needs a tester round or a trial, not edits. Nothing here is applied.

Supersedes the forward halves of #18 and #24 (both closed). #19 (recording/compaction) stays separate — it is tooling, not spec/guide.

A. Candidates for the next promotion pass

Ordered by strength of evidence.

  1. Harness-known identities are injected, never model-guessed. The agent-memory spec has this as an R11 sentence; the guide does not. Evidence it is needed: a clean-room regen against the current guide (regen-D) contains zero such language, so a spec generated today ships the same gap that made AC21 fail — the model, never told its thread id, guessed "current_thread" and the summarize tool honestly consolidated nothing. Second datapoint: the first wording named one tool, and an independent trial build applied it to exactly that tool, leaving summary expansion unscoped. Write it concept-anchored (identities the harness knows are injected and absent from the model-facing schema; identifiers the model legitimately holds stay model-supplied).

  2. A shipped default that no acceptance criterion exercises (D10 augmentation). Tool-description augmentation is ON by default, but the offline stub cannot return augmentation JSON and no live AC registers through a real model — so 2 of 2 builds shipped it unexercised. Compare R5, which the spec deliberately made stub-testable. Candidate rule: a Ledger default that changes behavior needs at least one AC that exercises that default, offline-testable or explicitly live.

  3. An AC depending on a fixture the spec never declares. AC7 and AC11 require a "scripted summarizer stub"; §5 declares four stubs and that is not one of them — in a section that opens "MUST create these fixtures exactly as described." 2 of 2 builds invented it, so part of the offline oracle rests on builder-authored fixture content. Candidate: §14 audit line — every fixture an AC names is declared in §5.

  4. Contract constraints with no AC behind them. SummaryRecord.description minLength 8 and R6's 8–12-word label band are stated in prose/schema and never asserted. Candidate: the existing rules↔ACs bidirectional audit extends to schema constraints that encode behavior.

  5. Prose-economy sweep. This batch grew the guide 11,041 → 12,313 words (+11.5%) and the spec 9,773 → 10,528 (+7.7%), concentrated in §0/§6.0 — the sections three testers called too dense. At this rate the guide doubles roughly every six batches. Candidate: a pass whose goal is subtraction (merge overlapping rules, move rationale out of emitted templates, delete restated mechanics) rather than new rules, with the regen properties as the safety net.

  6. F8 — fixture-corpus churn across regenerations. Parked by owner decision, carried here so it is not lost. Each regeneration authors fresh synthetic fixtures by design, so fixture identities churn and cross-generation comparisons carry noise. Not currently biting: with the Option-B pattern (surgical spec edits, no regen adoption) regenerations are evidence, not products.

B. Measurement watchlist

Needs a tester round or a disposable-clone trial. No edits pending.

  • Does the orientation block cure the density complaint? Three testers found the spec overwhelming up front (R16: "a brief, high-level overview at the top"; R18: "I needed a better explanations and instructions in the beginning"). The fix shipped in #25 and is unproven with humans — a build agent reusing the prose is not evidence.
  • Does the one-question gate feel light enough? R17 objected to "the heavy question-tool ceremony", and F12 replaced 13 questions with one for the default path. Needs a human reaction, not a probe.
  • Baseline resource benchmark (wall-clock, tokens, disk) for a recommended-baseline build. This is the answer to the cost-abandonment feedback, which the owner decided to measure rather than promise in spec text.
  • Does a build actually write resolved-decisions.md? Accepted residual from #25: the requirement is validated in text, the behavior is not. Rides the next trial rather than a paid probe build.
  • Ingestion idempotency — one tester hit UNIQUE errors re-running setup. The current build cannot crash that way (generated ids) but duplicates silently. Proposal recorded, unapplied: an idempotent-seed AC or an explicit "re-seeding duplicates; reset first" note.
  • arXiv 429 recurrence — transient rate limits hit a tester and our own runs; the client retries internally and R11 makes a mid-loop 429 non-fatal. Proposal recorded, unapplied: document transient-429 as expected for live (net) criteria rather than mandate backoff code.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Treat this as a planning and measurement watchlist rather than a single starter task. Read PR #25, the agent-memory spec and guide, then compare regen-D, AC7/AC11, D10, and the §5/§14 audits. A follow-up is ready when one candidate has a bounded edit and validation plan; watchlist items require a tester round or disposable-clone trial.

Written by the indexing model from the issue text.

Assessment

Domain
documentation, testing
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.