ai-cfia / ai-cfia/fertiscan

Split compliance prompt into system instructions + typed evaluation context

Aberta
#674 0 comentários 0 reações 0 responsáveis Ver no GitHub
backend
Linguagem predominante
HTML
Estrelas
5
Forks
2
Merge médio
1d 7h
PRs com merge (30d)
1

Descrição

# Context

The compliance evaluation prompt at `backend/app/prompt_templates/compliance_verification.md` is currently rendered as a single Jinja template that interleaves static instructions (Role, Verification Protocol, Constraints) with dynamic per-requirement data. The dynamic data is assembled by five formatter helpers in `backend/app/services/compliance.py` that each emit a bullet-list string for the template:

- `get_requirement_dictionary`
- `get_general_exemptions`
- `get_exemptions`
- `get_applicability_conditions`
- `get_requirement_provisions` (which also tacks on `guidance_en` as a trailing line)

Two consequences:
- The static instructions are re-rendered into the user message on every evaluation, so prompt caching never sees a stable prefix.
- Per-requirement `guidance_en` is appended at the bottom of the Requirement section and is regularly ignored by the model — e.g. requirements whose guidance says "flag as INCONCLUSIVE if no registration number" are returning definitive non-compliance verdicts instead.

# Outcome

- Static prompt content (Role, Verification Protocol, Constraints) is sent as a system message and is byte-stable across every requirement evaluation.
- Per-call legal context and label data are delivered to the model as a single typed object, replacing the per-section string formatters.
- Per-requirement guidance is a first-class field that the verification protocol explicitly references and applies.
- Jinja-based prompt rendering is removed; the prompt becomes a plain markdown file loaded as text.

# Acceptance criteria

- The Role / Verification Protocol / Constraints content is delivered as a system message that is identical across every call to `evaluate_requirement`.
- The dynamic inputs — dictionary, general exemptions, exemptions, applicability conditions, requirement provisions, per-requirement guidance, and label data — are delivered as a single structured payload (Pydantic model serialized to JSON).
- The verification protocol's vocabulary matches the field names in the structured payload — no mismatch between section names referenced in the protocol (e.g. "Dictionary") and the JSON keys delivered to the model.
- The five formatter helpers listed above are removed from `services/compliance.py`.
- Jinja-based prompt rendering is removed: `render_prompt()` in `services/compliance.py`, the `prompt_template_env` computed property in `config.py`, and any settings exclusively supporting prompt templating are deleted. Email template rendering is unaffected.
- Per-requirement guidance reaches the model in a way the verification protocol acknowledges, such that a requirement whose guidance directs an INCONCLUSIVE verdict under specific conditions produces INCONCLUSIVE when those conditions are met.
- List ordering within the structured payload is deterministic across runs.
- Existing tests covering `evaluate_requirement` continue to pass, with updates only where the payload shape demands them.

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.