Thinklab-SJTU / Thinklab-SJTU/Bench2Drive
[Question] High variance in evaluation results for the same model on identical route (id="2091")
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 144
- PR merge metrics
- No merged PRs in 30d
Description
Describe the question
Hi team,
I am encountering a significant inconsistency when evaluating my model. I ran the exact same model twice on the same case, but the results are completely different.
Test Case Details:
- Route ID:
2091 - Town:
Town12
Evaluation Results:
- Run 1:
score_composed= 21.14 (Failed) - Run 2:
score_composed= 100.0 (Completed)
Questions
- Is this level of variance normal for this evaluation framework?
- Is the randomness in the environment (e.g., NPC behaviors, traffic manager, spawning) expected to cause such a huge gap between two runs?
- Are there any specific configurations or random seeds I should fix to ensure reproducible evaluation results?
Thanks in advance for your help!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing route 2091 in Town12 twice with the same model and evaluation configuration. Check whether NPC behavior, traffic-manager behavior, spawning, or random seeds differ between runs; done means identifying whether the variance is expected and documenting the configuration needed for reproducible results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- autonomous-driving, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100