Add ability for partial reevaluation
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1k
- Forks
- 412
- Avg merge
- 6d 10h
- Merged PRs (30d)
- 3
Description
It often happens that if there is a bug in the test data, it only affects one or a small number of test cases. However, at the moment the only option is to reevaluate all test cases when this occurs, which may be very slow for large data sets.
This issue outlines my intention to implement partial reevaluation, where an admin can select some subset of testcases to be re-evaluated. Combined with task versions, this makes it much faster to test small changes to test data.
I plan to spend some time on this in April (unless somebody else beats me to it!)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by tracing the existing full test-case reevaluation flow and the admin interface that triggers it, then define how an administrator selects a subset and how task versions interact with that selection. Done means selected test cases can be reevaluated without rerunning unrelated cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100