forrtproject / forrtproject/flora-extractor
Prompt backlog from audit annotations (unapplied comment decisions)
- Dominant language
- Python
- Stars
- 2
- Forks
- 1
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 4
Description
The screening-audit comment layer carries a set of prompt decisions and instructions that were never executed (the applied ones landed in #89/#104/#105/#108). Collected here so they stop living in a comment sidebar. Most should ride with or after the §14 screening-prompt evaluation (see the current-state doc) so behaviour changes are measured, not assumed.
**Identification / resolution prompts**
- [ ] Remove the `CITED PATTERN` hint (F2) and measure (`shared/prompts.py:180`; scaffolding in `scratch_cited_pattern_*`). Audit D16 side-effect: F3 wraps non-anchor hints in "a human reviewer marked the previous answer as INCORRECT".
- [ ] Test removing the `CITED AUTHOR-YEAR PATTERNS` block on ~200 `llm_cited_candidates` rows, on OpenAI and Gemma — fold into the §14.6 evaluation sampling.
- [ ] Return explicit candidate keys (e.g. `smith2009`) instead of positional numbers in the identification/target JSON.
- [ ] Drop the F3 validator-feedback wrapper — a validator who spots a wrong target should name the right one.
- [ ] F1 FLoRA anchor note: decide — exclude FLoRA-validated DOIs at search time instead, and run FLoRA-validation as its own mode rather than in-band anchoring.
- [ ] Landing-page HTML substitute: keep or drop; if kept, stop labelling its resolutions `llm_fulltext`.
- [ ] Truncation constants (abstract 700 / intro 600 / methods 400): shorten deliberately or validate; make pipeline reporting reflect them.
- [ ] Title-pattern hint: measure value or drop.
**Outcome prompts**
- [ ] Move `GENUINE_ATTEMPT_RULE` above the outcome rules (currently appended after, `shared/prompts.py:631`), with the ~50-row before/after spot-check the audit called for.
- [ ] Merge the fulltext escalation with the abstract prompt — one prompt, no call history.
**All prompts**
- [ ] Static instructions first, per-row inputs last (prefix caching + instruction following).
- [ ] Standardise JSON block formatting, response keys, and the confidence vocabulary (`OUTCOME_ENUM` and the provider system message are already shared; renames are cheap since #104's content-complete cache keys auto-invalidate).
**Stage 2 filter prompt**
- [ ] Consolidate the phrase / no-phrase `needs_review` paths — one decision (is it a replication + what is the target), one path. The Stage-3 front door re-confirms every survivor since #104, so the split only costs Stage 2.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with shared/prompts.py and the current-state document, especially the sections referenced for screening-prompt evaluation. Review the cited scratch_cited_pattern_* scaffolding and the llm_cited_candidates sampling plan before changing prompts. Done means the selected decisions are implemented consistently, evaluated on the named samples, and reflected in pipeline reporting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, testing
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100