[RFC] Add a board for local runner to track the states of all tasks

Motivation

The local execution tasks (LocalRunner) of OpenCompass determine whether to enable debug mode by specifying the --debug flag. Since debug mode prints all logs of the tasks executed by the runner, it causes excessive on-screen log clutter. Therefore, debug mode is disabled during regular execution. Without debug mode, the execution status of tasks can only be viewed through a progress bar at the granularity of the entire task, as shown below:

# Execution command
opencompass --models deepseek_r1_streaming hf_llama_7b  --datasets aime2024_gen gsm8k_gen

# Partial printed logs
11/17 17:07:14 - OpenCompass - INFO - Current exp folder: outputs/default/20251117_170714
11/17 17:07:14 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored.
11/17 17:07:14 - OpenCompass - INFO - Partitioned into 2 tasks.
launch OpenICLInfer[DeepSeek-R1-0528/gsm8k,DeepSeek-R1-0528/aime2024] on CPU                                                                                                                                                    
launch OpenICLInfer[llama-7b-hf/gsm8k,llama-7b-hf/aime2024] on GPU 0                                                                                                                                                            
 50%|██████████████████████████████████████████████████████████████████████████████████████████████▌                                                                                              | 1/2 [01:01<01:01, 61.31s/it

In non-debug mode, the printed task information is insufficient. When multiple tasks are executed in parallel, it is impossible to monitor the execution status of all tasks simultaneously. If some tasks fail, their logs are easily overwritten by other outputs.

Proposed Change

We aim to introduce a status dashboard that can monitor the execution status of all ongoing tasks when running in the local environment (within LocalRunner).

The status dashboard should include the following information:

1. Task Name: Consistent with the task names printed by OpenCompass (e.g., OpenICLInfer[DeepSeek-R1-0528/gsm8k]).
2. Task Subprocess ID: Allows users to detect if the task subprocess has been killed.
3. Task Execution Progress: Displays the progress of inference or evaluation via a progress bar.
4. Task Execution Time: Enables intuitive tracking of how long each task has been running, helping identify task freezes.
5. Task Status: Marks the current phase of the task (e.g., "Model Weight Loading Phase").
6. Task Detailed Log Path: Consistent with the log paths printed by OpenCompass when errors occur (e.g., outputs/default/20251118_162917/logs/infer/DeepSeek-R1-0528/gsm8k.out).
7. Additional Extended Parameters: Supports extending other parameters, stored in a dictionary format.

Expected Display Effect of the Status Dashboard

+------------------------------------+-----------+-------------------------------------------------------+-------------+-------------+---------------------------------------------------+------------------------------------------------------+
| Task Name                          |   Process | Progress                                              | Time Cost   | Status      | Log Path                                          | Extend Parameters                                    |
+====================================+===========+=======================================================+=============+=============+===================================================+======================================================+
| vllm-api-general-chat/aime2024     |   1682636 | [##############################] 30/30 [30.0 it/s]    | 0:00:03     | finish      | logs/infer/vllm-api-general-chat/aime2024.out     | {'POST': 30, 'RECV': 30, 'FINISH': 30, 'FAIL': 0}    |
+------------------------------------+-----------+-------------------------------------------------------+-------------+-------------+---------------------------------------------------+------------------------------------------------------+
| vllm-api-general-chat/GPQA_diamond |   1682701 | [##############################] 198/198 [20.0 it/s]  | 0:00:06     | finish      | logs/infer/vllm-api-general-chat/GPQA_diamond.out | {'POST': 198, 'RECV': 198, 'FINISH': 198, 'FAIL': 0} |
+------------------------------------+-----------+-------------------------------------------------------+-------------+-------------+---------------------------------------------------+------------------------------------------------------+
| vllm-api-general-chat/gsm8k        |   1682634 | [##########                    ] 451/1319 [49.9 it/s] | 0:00:11     | inferencing | logs/infer/vllm-api-general-chat/gsm8k.out        | {'POST': 461, 'RECV': 451, 'FINISH': 451, 'FAIL': 0} |
+------------------------------------+-----------+-------------------------------------------------------+-------------+-------------+---------------------------------------------------+------------------------------------------------------+
| vllm-api-general-chat/drop         |   1682632 | [                              ] 270/9536 [49.9 it/s] | 0:00:11     | inferencing | logs/infer/vllm-api-general-chat/drop.out         | {'POST': 280, 'RECV': 270, 'FINISH': 270, 'FAIL': 0} |
+------------------------------------+-----------+-------------------------------------------------------+-------------+-------------+---------------------------------------------------+------------------------------------------------------+
| triton-api-general/aime2024        |           | NA                                                    | NA          | not start   |                                                   | None                                                 |
+------------------------------------+-----------+-------------------------------------------------------+-------------+-------------+---------------------------------------------------+------------------------------------------------------+
| triton-api-general/GPQA_diamond    |           | NA                                                    | NA          | not start   |                                                   | None                                                 |
+------------------------------------+-----------+-------------------------------------------------------+-------------+-------------+---------------------------------------------------+------------------------------------------------------+
| triton-api-general/gsm8k           |           | NA                                                    | NA          | not start   |                                                   | None                                                 |
+------------------------------------+-----------+-------------------------------------------------------+-------------+-------------+---------------------------------------------------+------------------------------------------------------+
| triton-api-general/drop            |           | NA                                                    | NA          | not start   |                                                   | None                                                 |
+------------------------------------+-----------+-------------------------------------------------------+-------------+-------------+---------------------------------------------------+------------------------------------------------------+

Architecture Design of the Status Dashboard

As shown in the diagram above:

• Before LocalRunner starts tasks, a separate thread is launched to run the task state monitor, which monitors all tasks.
• In the subprocess started by LocalRunner, a separate thread is launched to run the state collector before initializing the task instance. This collector captures the execution status of the task instance throughout its entire lifecycle.
• The task state monitor and state collector communicate task status via temporary files (intended to be in JSON Lines format) corresponding to each task. The state collector appends task status to the temporary file of each task at fixed intervals, while the task state monitor reads the latest data from all task temporary files at fixed intervals.

Class Diagram for the Changes

As shown in the diagram above:

• A new member variable task_states_monitor is added to LocalRunner. In the LocalRunner.launch method, an independent thread is used to call task_states_monitor.launch_state_board to start the status monitor (not launched in debug mode).
• A new member variable state_collector is added to BaseTask, which is initialized by passing an instance via the newly added init_state_collector method.
• Initialization of TaskStatesCollector requires passing a temporary file path (for communication with task_states_monitor), which is passed via the command line of the task file.
• The TaskStatesCollector.update_task_state method is used to update the task status during the execution of the BaseTask instance.
• The state_collector can also be further passed down to the OpenICL layer within the BaseTask instance.

Command Line Changes

• For task files that support independent launch via the command line (openicl_infer.py, openicl_attack.py, openicl_eval.py, subjective_eval.py), a new command line parameter --state-tmp-file needs to be added. This parameter passes the temporary file path for transmitting status information of each task. If no path is specified, TaskStatesCollector will not be launched in the task, ensuring no impact on other runners calling this task.
• When task_states_monitor is launched in LocalRunner, the --state-tmp-file /path/to/tmp/file flag must be added to the locally launched task command constructed as cmd = task.get_command(cfg_path=param_file, template=tmpl).

Feedback Period

1~2 weeks

CC List

Any Other Things

Will you implement it?

• I would like to implement this feature and create a PR!