aws-samples / aws-samples/hpc-cost-simulator

[FEATURE] Relax 'SlurmLogParser' dependency on AWS

Open
#100 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
14
Forks
5
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**
My search for a robust Slurm `sacct` log parser led me to this project. But `SlurmLogParser` is tied to a specific structure as defined in `SchedulerJobInfo` - it must have those fields, and any other fields are discarded. Would be good to support parsing almost any `sacct` output, only requiring a small core set of fields like `JobID` and `State`.

**Describe the solution you'd like**
I am happy to implement, but will involve significant refactoring so wanted to check first that such a change would be accepted. I expect the following specific changes:
- ~~rename `SchedulerJobInfo` to `AWSJobInfo`, to distinguish between any Slurm job and the set of fields your modelling needs~~
- ~~accordingly, rename `parse_job[s]` to `parse_aws_job[s]`~~
- ~~repurpose `parse_job` for parsing any `sacct` output, which requires first line be header, and uses header to as column names in output file~~

Update: that amount of refactoring not necessary, output CSV field names can be taken from input CSV header. I'll post a PR soon.

Contributor guide

Open the contributing guide

Research direction

Start by reading SlurmLogParser, SchedulerJobInfo, and the parse_job/parse_jobs entry points described in the issue. Trace how sacct headers and fields become CSV output, then verify that arbitrary input columns are preserved while a small core including JobID and State remains supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.