mlcommons / mlcommons/inference
[Llama3-8b-edge] Singlestream performnace mode samples
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 650
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 6
Description
@arjunsuresh @pgmpablo157321 While giving a run of the singlestream performance mode, I noticed that the number of samples being run is much higher than the 5k that was designated by the taskforce. I'm tracking the number of samples being received and it's gone beyond even 10k.
The log details also has some weird values for the generated and effective min query count (I only used the default mlperf.conf parameters)
:::MLLOG {"key": "qsl_reported_total_count", "value": 5000, "time_ms": 0.027256, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 1198, "pid": 3313545, "tid": 3313545}}
:::MLLOG {"key": "qsl_reported_performance_count", "value": 5000, "time_ms": 0.027256, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 1199, "pid": 3313545, "tid": 3313545}}
:::MLLOG {"key": "requested_min_query_count", "value": 100, "time_ms": 0.035443, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 318, "pid": 3313545, "tid": 3313545}}
:::MLLOG {"key": "requested_max_query_count", "value": 0, "time_ms": 0.035443, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 319, "pid": 3313545, "tid": 3313545}}
:::MLLOG {"key": "requested_performance_sample_count_override", "value": 5000, "time_ms": 0.035443, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 337, "pid": 3313545, "tid": 3313545}}
:::MLLOG {"key": "effective_min_query_count", "value": 125000, "time_ms": 0.036073, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 431, "pid": 3313545, "tid": 3313545}}
:::MLLOG {"key": "effective_max_query_count", "value": 0, "time_ms": 0.036073, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 432, "pid": 3313545, "tid": 3313545}}
:::MLLOG {"key": "effective_min_sample_count", "value": 125000, "time_ms": 0.036073, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 433, "pid": 3313545, "tid": 3313545}}
:::MLLOG {"key": "effective_performance_sample_count", "value": 5000, "time_ms": 0.036073, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 451, "pid": 3313545, "tid": 3313545}}
:::MLLOG {"key": "generated_query_count", "value": 125000, "time_ms": 17.024346, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 427, "pid": 3313545, "tid": 3313545}}
This is an unexpected behavior.
Below is the last processed queries I've tracked so far
Processed: 10100 queries
Processed: 10150 queries
Processed: 10200 queries
Processed: 10250 queries
Processed: 10300 queries
Processed: 10350 queries
Processed: 10400 queries
Processed: 10450 queries
Processed: 10500 queries
Processed: 10550 queries
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the Llama3-8b-edge singlestream performance mode with the default mlperf.conf and inspect the MLPerf LoadGen settings shown in the logs. Compare the configured 5,000 samples with effective_min_query_count and generated_query_count, then confirm that processing stops at the intended sample count.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- machine-learning
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100