mlcommons / mlcommons/inference

[Llama3-8b-edge] Singlestream performnace mode samples

Open
#2,284 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.6k
Forks
650
Avg merge
1d 22h
Merged PRs (30d)
6

Description

@arjunsuresh @pgmpablo157321 While giving a run of the singlestream performance mode, I noticed that the number of samples being run is much higher than the 5k that was designated by the taskforce. I'm tracking the number of samples being received and it's gone beyond even 10k.

The log details also has some weird values for the generated and effective min query count (I only used the default mlperf.conf parameters)

:::MLLOG {"key": "qsl_reported_total_count", "value": 5000, "time_ms": 0.027256, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 1198, "pid": 3313545, "tid": 3313545}}
:::MLLOG {"key": "qsl_reported_performance_count", "value": 5000, "time_ms": 0.027256, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 1199, "pid": 3313545, "tid": 3313545}}
:::MLLOG {"key": "requested_min_query_count", "value": 100, "time_ms": 0.035443, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 318, "pid": 3313545, "tid": 3313545}}
:::MLLOG {"key": "requested_max_query_count", "value": 0, "time_ms": 0.035443, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 319, "pid": 3313545, "tid": 3313545}}
:::MLLOG {"key": "requested_performance_sample_count_override", "value": 5000, "time_ms": 0.035443, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 337, "pid": 3313545, "tid": 3313545}}
:::MLLOG {"key": "effective_min_query_count", "value": 125000, "time_ms": 0.036073, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 431, "pid": 3313545, "tid": 3313545}}
:::MLLOG {"key": "effective_max_query_count", "value": 0, "time_ms": 0.036073, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 432, "pid": 3313545, "tid": 3313545}}
:::MLLOG {"key": "effective_min_sample_count", "value": 125000, "time_ms": 0.036073, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 433, "pid": 3313545, "tid": 3313545}}
:::MLLOG {"key": "effective_performance_sample_count", "value": 5000, "time_ms": 0.036073, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 451, "pid": 3313545, "tid": 3313545}}
:::MLLOG {"key": "generated_query_count", "value": 125000, "time_ms": 17.024346, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 427, "pid": 3313545, "tid": 3313545}}

This is an unexpected behavior.
Below is the last processed queries I've tracked so far

Processed: 10100 queries
Processed: 10150 queries
Processed: 10200 queries
Processed: 10250 queries
Processed: 10300 queries
Processed: 10350 queries
Processed: 10400 queries
Processed: 10450 queries
Processed: 10500 queries
Processed: 10550 queries

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the Llama3-8b-edge singlestream performance mode with the default mlperf.conf and inspect the MLPerf LoadGen settings shown in the logs. Compare the configured 5,000 samples with effective_min_query_count and generated_query_count, then confirm that processing stops at the intended sample count.

Written by the indexing model from the issue text.

Assessment

Tech stack
machine-learning
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.