alibaba / alibaba/clusterdata

[gpu-v2026] Is job_execution_summary only supported from around Day 109?

Open
#259 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2.2k
Forks
482
PR merge metrics
No merged PRs in 30d

Description

Hi ASI Trace Team,

Thank you for releasing the Alibaba Cluster Trace GPU v2026 dataset.

We are using the trace for task-resolved workload modeling and trace-driven GPU cluster scheduling. During our audit, we observed a very clear temporal boundary when linking `asi_opensource_pod_hourly` with `asi_opensource_job_execution_summary`.

Using only pod-ID coverage, before applying any Duration threshold or matching score, we obtain:

- Day 4–108: only about 0.2685% of reconstructed pod execution episodes have a pod ID that appears in `job_execution_summary`.
- Day 109–129: the coverage becomes 100%.
- Among our reliably linked execution episodes, 99.7845% come from Day 109 onward.
- Most of the small number of matches before Day 109 are long-running episodes that continue across the Day-109 boundary.

This suggests that the temporal boundary exists in the released data itself, rather than being caused by our execution matching procedure.

We noticed that Issue #257 also asks about the missing submission timestamp in `job_execution_summary`, and Issue #258 reports that `state_public` is mostly `Unknown` during roughly the first 100 days. Our observation may be related to the same metadata-coverage boundary.

Could you please clarify:
1. Does `asi_opensource_job_execution_summary` intentionally cover only executions from approximately Day 109 onward?
2. If so, is there any execution-summary data for Day 0–108 that could potentially be released?
3. Is there any anonymized field, mapping table, or execution-instance ID that can associate a row in `job_execution_summary` with a relative `day` / `hour` in `pod_hourly`?
4. How exactly is `duration_hours` defined? In particular, does it represent wall-clock execution span, active allocated execution time, or another quantity?
5. Would it be possible to release the upstream script used to construct `job_execution_summary`, or relative start/end timing information?

We do not need calendar dates, user identities, raw pod names, or any business-sensitive metadata. Relative day/hour information or a stable anonymized execution ID would be sufficient for our research.

We would be happy to share our audit scripts and aggregate linkage statistics if useful.

Thank you very much for releasing this valuable dataset.

Contributor guide

No contributing guide indexed for this repository

Research direction

Review Issues #257 and #258 alongside the documented pod_hourly and job_execution_summary fields. Compare the reported Day 4–108 and Day 109–129 coverage, then determine whether the dataset documentation can answer the coverage, timing, linkage, and duration questions. Done means the release documentation or maintainer response clearly states the supported period and available associations.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.