apache / apache/beam

Performance Regression or Improvement: test_cloudml_benchmark_criteo_10GB-runtime_sec:runtime_sec

Open
#39,667 2 comments 0 reactions 0 assignees View on GitHub
awaiting triage perf-alert
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Performance change found in the
test: `test_cloudml_benchmark_criteo_10GB-runtime_sec` for the metric: `runtime_sec`.

For more information on how to triage the alerts, please look at
`Triage performance alert issues` section of the [README](https://github.com/apache/beam/tree/master/sdks/python/apache_beam/testing/analyzers/README.md#triage-performance-alert-issues).

`Test description:` TFT Criteo test on 10 GB data.
Test link - https://github.com/apache/beam/blob/42d0a6e3564d8b9c5d912428a6de18fb22a13ac1/sdks/python/apache_beam/testing/benchmarks/cloudml/cloudml_benchmark_test.py#LL104C7-L104C41

```

timestamp: Wed Aug 5 23:21:10 2026, metric_value: 2800.29
timestamp: Tue Aug 4 23:22:17 2026, metric_value: 2808.33
timestamp: Mon Aug 3 23:37:30 2026, metric_value: 2774.47
timestamp: Sun Aug 2 23:07:44 2026, metric_value: 2729.92
timestamp: Sat Aug 1 23:09:53 2026, metric_value: 2732.18
timestamp: Fri Jul 31 23:09:55 2026, metric_value: 2486.15
timestamp: Thu Jul 30 23:08:38 2026, metric_value: 2467.28
timestamp: Wed Jul 29 22:53:13 2026, metric_value: 2283.78
timestamp: Tue Jul 28 23:26:00 2026, metric_value: 2327.85
timestamp: Mon Jul 27 23:07:24 2026, metric_value: 2777.71
timestamp: Sun Jul 26 23:03:46 2026, metric_value: 2858.59 <---- Anomaly
timestamp: Sat Jul 25 22:54:07 2026, metric_value: 2398.88
timestamp: Fri Jul 24 23:00:38 2026, metric_value: 2379.31
timestamp: Thu Jul 23 23:14:49 2026, metric_value: 2403.32
timestamp: Wed Jul 22 22:56:31 2026, metric_value: 2223.12
timestamp: Tue Jul 21 22:56:42 2026, metric_value: 2191.53
timestamp: Mon Jul 20 22:55:29 2026, metric_value: 2081.13
timestamp: Sun Jul 19 22:49:13 2026, metric_value: 2286.14
timestamp: Sat Jul 18 22:48:21 2026, metric_value: 2334.37
timestamp: Fri Jul 17 22:44:25 2026, metric_value: 2140.38
timestamp: Thu Jul 16 22:48:39 2026, metric_value: 2199.59

```

Contributor guide

Open the contributing guide

Research direction

Start with the Triage performance alert issues section in sdks/python/apache_beam/testing/analyzers/README.md, then inspect the linked test_cloudml_benchmark_test.py entry point around line 104. Compare the runtime history with the reported anomaly and follow the documented triage process. Done means the alert is classified as a real change or noise, with its cause and validation recorded.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.