Inquiry about evaluation
Nobody has claimed this yet.
- Dominant language
- HTML
- Stars
- 708
- Forks
- 32
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 5
Description
Thank you for the good work!
I have some questions regarding the evaluation of MOSS-VL-Realtime on streaming benchmarks, particularly on the proactive tasks of StreamingBench and OVO-Bench. Since the two benchmarks both evaluate a model's proactivity in a pseudo-streaming manner (OVO-Bench queries the model at several preset time points, while StreamingBench polls the model every second and terminates upon the first positive response. Both of them do not actually let the model make the decision to response or remain silent by itself.), so I am wondering if you have followed the established evaluation protocals of these two benchmarks, or make some alterations to align them with real streaming settings.
If possible, could you please open-source your evaluation code? That would be a great help to me! Thank you again for your impressive work!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the evaluation protocols for StreamingBench and OVO-Bench described in the issue, then read the discussion for any maintainer response. Done would require a clarified account of the protocols used and, if agreed, the evaluation code being made available.
Written by the indexing model from the issue text.
Assessment
- Domain
- machine-learning, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100