RFC: Structured data for executors to report on Job result

AI Disclaimer: This was conceptualized and written with the assistance of AI (Claude). I have reviewed this substantially.

Related issues/PRs:

• snakemake#2441 — More control on retry behaviour
• snakemake#2123 — Cluster retry get failure reason
• snakemake#226 — Keep track of external job IDs after completion
• slurm#65 — Dynamic resources depending on retry reason

Background:

When a remote job fails, the executor passes the result as a pre-formatted string in report_job_error(j, msg=msg). The information exists on the executor's side but gets flattened into display text before it reaches core. A logger plugin, or snakemake itself, can't programmatically distinguish OOM from timeout, can't access numeric resource usage, can't know where the job ran. External job IDs aren't persisted either, so users can't even go back and query their scheduler after the fact (snakemake/snakemake#226).

This blocks failure-aware retries — probably the most impactful consequence. Resource functions receive an attempt counter but have no way to know why the previous attempt failed, so users blindly increase both memory and time on every retry, wasting cluster resources and queue time. (snakemake/snakemake#2441, snakemake/snakemake#2123, snakemake/snakemake-executor-plugin-slurm#65)

Proposal

JobResult is a new dataclass in snakemake-interface-common that executors populate with structured execution results. The design is executor-agnostic — any executor plugin populates it with whatever data it can obtain from its backend.

@dataclass
class JobResult:
    status: JobResultStatus = JobResultStatus.UNKNOWN
    exit_code: Optional[int] = None
    error_type: Optional[str] = None       # freeform platform-specific state, e.g. "OUT_OF_MEMORY"
    error_message: Optional[str] = None
    external_job_id: Optional[str] = None  # SLURM job ID, k8s pod name, etc.
    environment: Optional[ExecutionEnvironment] = None
 
@dataclass
class ExecutionEnvironment:
    hostname: Optional[str] = None
    node_name: Optional[str] = None
    partition: Optional[str] = None

Everything is Optional — executors just fill in what they can. None means "not available." external_job_id addresses snakemake/snakemake#226— persisting the scheduler's job ID so users can query resource usage after the fact or include it in reports.

The flow

Using SLURM as an example (the design is executor-agnostic):

1. Executor queries its backend when a job reaches a terminal state (e.g. sacct with extended format).
2. Stashes JobResult on SubmittedJobInfo.aux["job_result"].
3. RemoteExecutor.report_job_success/error propagates it onto the Job object.
4. The scheduler reads it when building the JobEvent (see logging redesign).
5. Formatters and plugins access typed fields.

Uses the existing aux dict — no executor interface signature changes. Old executors that don't set it just produce None.

JobResult goes in snakemake-interface-common because both executor and logger interface packages need to reference it without creating a dependency edge between them. Snakemake core itself will also import it.

Failure mode enum

JobResultStatus encodes retry semantics, not platform-specific states. The enum answers: what should snakemake do differently on retry?

class JobResultStatus(Enum):
    SUCCESS = "success"
    FAILED = "failed"          # generic failure — retry with same resources
    TIMEOUT = "timeout"        # wall time exceeded — retry with more time
    OOM = "oom"                # memory exceeded — retry with more memory
    PREEMPTED = "preempted"    # preempted/evicted/node failure — just retry as-is
    CANCELLED = "cancelled"    # user or system cancelled — don't retry
    UNKNOWN = "unknown"        # couldn't determine — treat as FAILED

Platform-specific states go in the freeform error_type string on JobResult. Each executor maps its own vocabulary to these categories. This enum is the contract for future failure-aware retry logic — when core eventually implements "retry with adjusted resources based on failure mode" (snakemake/snakemake#2441, snakemake/snakemake#2123), it switches on these categories.

Would appreciate any thoughts on this!

Start by reading the proposed JobResult and ExecutionEnvironment dataclasses for snakemake-interface-common, then trace SubmittedJobInfo.aux["job_result"] through RemoteExecutor.report_job_success/error and JobEvent construction. Review logging_redesign.md and related issues to clarify the contract; done means the structured result can be referenced by core, formatters, and plugins without executor interface changes.