NatLabRockies / NatLabRockies/COMPASS
Confidence metric and/or self-repair
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 17
- Forks
- 4
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 17
Description
Consider something like this: after extraction has been completed. Go through every extracted feature (maybe even every non-extracted feature?) and ask the model to provide supporting evidence in the extraction text. Come up with a rating system of confidence based on that (maybe even a self-fix option)
Ideas from discussion:
Ask the model to differentiate evidence at multiple level along the lines of "mentions the evidence keywords," "evidence can be inferred," "direct evidence in the text" with additional levels specific to questions.
Can also use decision-tree-like process to evaluate
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by locating where extraction completes and where the model is queried, then review how extracted features and evidence are represented. Done should include an agreed confidence scale, evaluation of evidence levels, and a defined decision on whether self-repair is included.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100