Need solution to get target # of ratings per hit
Nobody has claimed this yet.
- Dominant language
- HTML
- Stars
- 230
- Forks
- 61
- PR merge metrics
- No merged PRs in 30d
Description
The current implementation may not give the requested number of ratings per hit because hits may fall into the "accept but not use". In a recent run 34% of hits fell into this bucket. This adds inconsistency to the ratings (not a constant number) and is also inefficient (wasted $). There are at least two solutions:
- Add a real-time failure for the pair comparison environment check which was 72% of failures in a recent run.
- Use the MTurk API to create a controller that monitors and updates the hits to meet the target goals.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are mentioned. Start by investigating the accept-but-not-use failures and the pair comparison environment check, then review the Amazon Mechanical Turk integration. Done should mean the requested number of usable ratings per hit is met consistently without unnecessary spending.
Written by the indexing model from the issue text.
Assessment
- Domain
- api, backend, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100