OpenMOSS / OpenMOSS/MOSS-VL

Inquiry about evaluation

Open
#22 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
HTML
Stars
708
Forks
32
Avg merge
1d 18h
Merged PRs (30d)
5

Description

Thank you for the good work!

I have some questions regarding the evaluation of MOSS-VL-Realtime on streaming benchmarks, particularly on the proactive tasks of StreamingBench and OVO-Bench. Since the two benchmarks both evaluate a model's proactivity in a pseudo-streaming manner (OVO-Bench queries the model at several preset time points, while StreamingBench polls the model every second and terminates upon the first positive response. Both of them do not actually let the model make the decision to response or remain silent by itself.), so I am wondering if you have followed the established evaluation protocals of these two benchmarks, or make some alterations to align them with real streaming settings.

If possible, could you please open-source your evaluation code? That would be a great help to me! Thank you again for your impressive work!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the evaluation protocols for StreamingBench and OVO-Bench described in the issue, then read the discussion for any maintainer response. Done would require a clarified account of the protocols used and, if agreed, the evaluation code being made available.

Written by the indexing model from the issue text.

Assessment

Domain
machine-learning, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.