andreagrandi / andreagrandi/draftomen

Define a fixed narration evaluation corpus

Đang mở
#453 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
draftomen enhancement size: S
Ngôn ngữ chính
Python
Star
0
Fork
0
Merge trung bình
1 giờ 23 phút
Pull request đã merge (30 ngày)
181

Mô tả

## Problem

Model quality cannot be judged by a few attractive examples or tests that merely check for nonempty output.

## Proposed change

Create fixed fact-contract fixtures and expected permitted claims covering all nine PRD situations: obvious best card, close decision, in-color strength, color deviation, low data, near-identical win rates, several strong alternatives, incomplete statistics, and multiple material factors. Include literal card names containing numbers or punctuation so numeric guards cannot confuse names with invented metrics. Define automated checks plus a human factuality/natural-English rubric. Keep this slice to corpus and evaluation specification; actual runtime benchmarking belongs to Establish local narration feasibility and delivery decisions.

## Acceptance Criteria

- [ ] AC1: A corpus coverage check maps every required situation to an immutable fixture with its engine-selected recommendation and permitted claims.
- [ ] AC2: Deliberately incorrect example outputs fail the rubric for invented statistics/reasons, altered numbers/colors, or contradicting the recommendation; verified by evaluator checks or recorded rubric review.
- [ ] AC3: Valid examples containing missing values, close-choice language, and numeric card names are judged correctly; verified by boundary examples.
- [ ] AC4: The evaluation specification separates deterministic format/number checks from reviewed semantic factuality and naturalness; verified by review of the rubric.

## Implementation classification

- **Estimated size:** S
- **Orchestration risk:** Normal
- **Reason:** A bounded fixture set and explicit quality rubric establish the evaluation baseline.

## Dependencies

Direct prerequisites: #452. Consumer: Establish local narration feasibility and delivery decisions.

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.