Evaluate public test cases with higher priority
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
Research direction
Start by reading JobGroup, EvaluationService, and SubmissionResult to understand how test cases become jobs and how evaluation_outcome and evaluation_tries are recorded. Define the expected behavior for prioritizing public cases and representing partially evaluated submissions; done means public results receive higher priority without losing coherent private-case evaluation state.
Written by the indexing model from the issue text.
Description
It could be useful to evaluate public test cases (and possibly private test cases on submissions with used tokens) with a higher priority: These are the results the contestants most eagerly await during the contest while for private test case results, larger delays may be acceptable.
In particular, contests with few workers would benefit from such a feature.
But even if the evaluation of the complete test case set is comparatively fast (few minutes) and there are many workers, this would be useful. It would give the contestants at least some feedback for submissions during the last minutes of the contest (e.g. to point out trivial mistakes like compile errors, left-over debugging output, selection of the wrong file, ...).
This would probably require changes to JobGroup (to generate separate jobs for public and private test cases), EvaluationService and SubmissionResult (to introduce some method to find out whether the submission has only been partially evaluated) at least. It is not clear to me what to do with SubmissionResult.evaluation_outcome and SubmissionResult.evaluation_tries (split it up or somehow share it across public/private test cases).
- Dominant language
- Python
- Stars
- 1k
- Forks
- 412
- Avg merge
- 6d 10h
- Merged PRs (30d)
- 3
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from cms-dev/cms
-
Difficulty 3/5 1-2 days Newbie friendliness 65/100
-
Difficulty 4/5 3-5 days Newbie friendliness 45/100
-
Difficulty 4/5 3-5 days Newbie friendliness 45/100
-
BatchAndOutput submissions should not require the source code to compile to grade the output fileas Open
Difficulty 3/5 1-2 days Newbie friendliness 62/100
-
Difficulty 3/5 1-2 days Newbie friendliness 55/100
Similar issues
-
area/auth bug comp/agent P3 platform/discord type/security
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
NousResearch/hermes-agent#117848 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
bancolombia/sentinel#23 ·
-
test md OpenCI
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
integration:quickjs org:external priority:backlog topic:code-interpreter topic:middleware type:feature
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
langchain-ai/deepagents#6450 ·
-
bug client
Difficulty 2/5 1-3 hours Newbie friendliness 88/100