sys-intelligence / sys-intelligence/system-intelligence-benchmark
Run tasks in SystemExams multiple times to calculate final score.
Open
Nobody has claimed this yet.
- Dominant language
- TLA
- Stars
- 55
- Forks
- 13
- PR merge metrics
- No merged PRs in 30d
Description
We find that different run of the SysemExam will have minor difference on performance. We may need to run each tasks multiple times and calculate the average score.
@tareknaser please help on this when you are testing SystemExam. Thanks.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the SystemExams task runner and the code that calculates final scores. Trace how individual task results are collected, then define completion as repeated runs producing an average score; verify the resulting benchmark output across multiple executions.
Written by the indexing model from the issue text.
Assessment
- Domain
- performance, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100