neurostuff / neurostuff/autonima
IDEA: Stricter screening * "unsure" categorization
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4
- Forks
- 1
- Avg merge
- 12h 56m
- Merged PRs (30d)
- 2
Description
A big issue with AI screening is that AI trends towards consensus
As such, it is unlikely to effectively implement new theoretical screening rules well, and instead default back to consensus framing.
This is problematic for literature reviews/meta-analysis, as the author may be trying to prove a novel point. Even if they are not, AI is likely to be overly confident in its categorization at times.
A possible solution is to allow the screening to categorize studies as "Unsure", in which case a human would perform the final review on a subset of studies.
Combined with stricter prompting (e.g. only categorize as this type of domain if its one of these tasks: [enumerate tasks]. If a task seem relevant but is not explicilty named, catevorize as "unsure").
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are identified in the issue. Start by locating where screening categories and prompts are defined, then determine how an explicit "Unsure" result and stricter task-based prompting would flow through the review process. Done should include a human-review path for unsure studies and tests covering the new categorization.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100