QuantEcon / QuantEcon/qeps

Process idea: field testing a QEP while in Draft

Open
#14 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

discuss qep
Dominant language
No language data
Stars
1
Forks
0
Avg merge
2d 5h
Merged PRs (30d)
7

Description

Idea

Add field testing to the QEP process: once a QEP is in Draft, run one or more real decisions against the draft text and record the outcome in the discussion PR, before the decision deadline. This would land as an in-place amendment to QEP-1's "How a QEP is decided" section if adopted.

Where the idea comes from

QEP-3 (#7) got two rounds of field testing during its comment window, and both were more informative than any amount of reading the text:

  • A validation. A live placement decision (project-reporting vs workspace-reporting) was run against the draft's goal-vs-fleet boundary rules, which decided it without interpretation. That is direct evidence the rules work in practice — exactly what a reviewer wants to know before accepting a standard, and something a read-through cannot establish.
  • A defect found while the draft was still cheap to change. Pulling on one thread the test exposed (reports-activity doesn't parse under the registry) revealed that the workflow-* type names a mechanism rather than a role, and produced a concrete restructure proposal (reporter-*/task-*) before acceptance rather than as a post-acceptance amendment.
  • An earlier field report on the same PR (the translation program's bench placement) surfaced two text gaps that were fixed in-flight (8150620).

The pattern generalises: a standard-type QEP exists to settle recurring decisions, so the natural test of a draft is to settle one real decision with it and see whether the text was sufficient. Every outcome is useful — a clean decision becomes validation evidence, a strained one becomes an amendment while amending is free.

Proposal sketch

Amend QEP-1's decision process with an optional step between Draft and Decide:

(Optional, encouraged for standards) Field test. Apply the draft to at least one real decision it claims to govern, and record the outcome as a field report comment on the discussion PR: what was decided, whether the text sufficed, and any amendments it motivated. Field reports are review evidence at the deadline.

Deliberately lightweight: no template, no gate, no new status — just a named, expected activity so authors and reviewers know it counts as review.

Points to decide

  1. Optional vs expected. Fully optional, or expected for type: standard QEPs (which govern recurring decisions and therefore always have a testable case) while remaining optional for process/informational QEPs?
  2. Interaction with the deadline. Leave the one-to-two-week window unchanged, or note that a substantive amendment arising from a field test should reset a short fresh window (as is happening on #7)?
  3. Who runs it. The author testing their own draft is still valuable (both #7 rounds were author-run), but a field test by a second maintainer is stronger evidence; worth encouraging without requiring.
  4. Naming. "Field test" / "field report" as used on #7, unless someone prefers another term.

If there's support, I'll draft the QEP-1 amendment (version bump per QEP-1's own rules) as a PR.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read QEP-1's "How a QEP is decided" section and compare the field reports from QEP-3 (#7), including commit 8150620. Resolve the four proposal points, then draft the QEP-1 amendment and required version bump if the project supports it. Done means the process change and its rationale are documented for review.

Written by the indexing model from the issue text.

Assessment

Domain
documentation
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.