HewlettPackard / HewlettPackard/FastSim
feat: run FastSim from a job trace + slurm.conf only (optional dumps, synthesized associations/QOS)
- Dominant language
- Python
- Stars
- 4
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
## What / Why
Running FastSim currently requires seven SLURM dumps, most of which need privileged access (`sacctmgr`, `sinfo`, `sreport` on a live cluster). Most researchers only have a **historical job trace** (e.g., from a database of job records) and can usually obtain the site's **slurm.conf**. This feature makes those two inputs sufficient to run the simulator, while every other dump remains usable when available.
## Design
**Required inputs:** `job_dump` (sacct-style pipe-delimited trace), `slurm_conf`, `considered_partitions`, `sim_start`, `sim_end`.
**Optional inputs** (each independently; used when provided, synthesized/empty when not):
| Input | When absent |
|---|---|
| `assocs_dump` | Synthesize a flat fairshare tree from the trace: root → one account per unique `Account` → one user per unique `(User, Account)`, all with **identical `Shares=1`** (users generally can't see real allocation awards). `Partition=NaN` so each user assoc covers all partitions (matches existing `_load_tree_slurm` semantics). No `MaxJobs`/`MaxSubmit` → no association limits. |
| `qos_dump` | Synthesize QOS entries for every QOS name referenced by the trace, plus every partition `QOS=` name from slurm.conf, plus `normal` (the partition default). Equal priority, all limits `None` → no QOS holds. |
| `node_events_dump` | Empty events frame → no down/drain events. |
| `resv_dump_current` / `resv_dump_historic` | Each independently optional; empty reservation frame when neither. `Queue._clean_reservations` already clears job reservation args not in `valid_resv`, so jobs that referenced reservations run as normal jobs (marked `ignore_in_eval`). |
**Job trace column tolerance:** `get_cleaned_job_df` gains a required-column check with a clear error (`JobID, Submit, Start, End, State, Partition, User, Account, ReqNodes, AllocNodes, Timelimit`) and tolerates missing optional columns: `QOS` (→ `normal`), `ConsumedEnergyRaw` (→ triggers existing zero-power fallback), `JobName`/`Reason` (→ `None`), `SubmitLine` (already optional).
**Key implementation points:**
- `config.py`: shrink `mandatory_fields`; add `None` defaults for the five optional dump paths, `[]` for `impromptu_reservation_names`, `"default"` for `system`, so the config namedtuple always has the attributes.
- `data_reader.py`: empty-frame paths for events/reservations; `synthesize_qos(names)`; `synthesize_assocs(df_jobs)` producing the same `User|Account|ParentName|Partition|Shares` frame `_load_tree_slurm` consumes.
- `fairshare.py`: `_load_tree_slurm` accepts a path or a prepared DataFrame.
- `controller.py`: pass QOS name universe to `get_qos` and assoc source (path or synthesized frame) to `FairTree`.
- Full-dump behavior must be byte-for-byte unchanged (verified against baseline via `scripts/compare_results.py`).
## Subtasks
- [x] Config: optional dump paths with defaults; shrunk mandatory set; `sim_start`/`sim_end` mandatory
- [x] data_reader: optional node events and reservation dumps (each reservation dump independently optional)
- [x] data_reader + controller: QOS synthesis when `qos_dump` absent
- [x] data_reader + fairshare + controller: association synthesis (flat tree, identical shares) when `assocs_dump` absent
- [x] data_reader: job-trace column tolerance (required-column validation + optional-column defaults)
- [x] Example minimal config + README documentation
- [x] Verification: trace-only run end-to-end; full-dump baseline regression (identical results); partial-data smoke run
Contributor guide
No contributing guide indexed for this repository
Research direction
Review config.py, data_reader.py, fairshare.py, and controller.py, then inspect the minimal config and README documentation. Run the trace-only and partial-data scenarios, and use scripts/compare_results.py to verify that full-dump results remain byte-for-byte identical; completion is covered by all listed subtasks and these checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100