AletheiaFact / AletheiaFact/agencia

Test claims for each label to check veracity

Open
#3 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2
Forks
0
PR merge metrics
No merged PRs in 30d

Description

### Background Information:
Our agent-based fact-checking system employs various classification labels to analyze and verify claims. To optimize the effectiveness and precision of this system, it is essential to assess and refine the agent's behavior and decision-making processes. This involves creating variations of claims for each classification label and implementing test cases. These efforts will help determine whether the current system needs adjustments in the prompts or if tasks should be distributed among additional agents to enhance performance and accuracy.

### What:
This task comprises two main objectives:
1. **Adapt the claim for each fact-checking label**: Modify the claim to fit all classification labels used by the fact-checking agents. This adaptation helps in understanding how well each label is functioning.

2. **Add claim test cases for each classification label**: Construct test cases that will serve as benchmarks for evaluating the performance of the classification labels. This will aid in identifying which labels are performing adequately and which may require modifications or further refinement after any system updates.

### How:
1. **Catalog existing classification labels**: Review all current classification labels in the system to understand their intended use and criteria.
`Not Fact`, `Trustworthy`, `Trustworthy, but`, `Arguable`, `Misleading`, `False`, `Unsustainable`, `Exaggerated`, `Unverifiable`

2. **Create claim variations**: For every classification label, adapt existing claims or generate new variations that clearly exemplify what the label is designed to detect or classify. Ensure these claims are representative of different styles, contexts, and complexities.

3. **Develop and implement test cases**: Formulate multiple test cases for each classification label that include a claim and the expected outcome. These cases should be diverse enough to challenge the agent’s decision-making capabilities under various scenarios.

#### Acceptance Criteria:
- Each fact-checking label must be associated with a claim variation that align with the label's definitions.
- Test cases per classification label must be developed, demonstrating the label's application in varied contexts.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.