aws-samples / aws-samples/hpc-cost-simulator
[FEATURE] Relax 'SlurmLogParser' dependency on AWS
- Dominant language
- Python
- Stars
- 14
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
**Is your feature request related to a problem? Please describe.**
My search for a robust Slurm `sacct` log parser led me to this project. But `SlurmLogParser` is tied to a specific structure as defined in `SchedulerJobInfo` - it must have those fields, and any other fields are discarded. Would be good to support parsing almost any `sacct` output, only requiring a small core set of fields like `JobID` and `State`.
**Describe the solution you'd like**
I am happy to implement, but will involve significant refactoring so wanted to check first that such a change would be accepted. I expect the following specific changes:
- ~~rename `SchedulerJobInfo` to `AWSJobInfo`, to distinguish between any Slurm job and the set of fields your modelling needs~~
- ~~accordingly, rename `parse_job[s]` to `parse_aws_job[s]`~~
- ~~repurpose `parse_job` for parsing any `sacct` output, which requires first line be header, and uses header to as column names in output file~~
Update: that amount of refactoring not necessary, output CSV field names can be taken from input CSV header. I'll post a PR soon.
Contributor guide
Research direction
Start by reading SlurmLogParser, SchedulerJobInfo, and the parse_job/parse_jobs entry points described in the issue. Trace how sacct headers and fields become CSV output, then verify that arbitrary input columns are preserved while a small core including JobID and State remains supported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100