Lightning-AI / Lightning-AI/sdk

bug(jobs): maxRunAttempts=1 reads back as 0

Open
#152 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
10
Forks
3
Avg merge
2d 1h
Merged PRs (30d)
32

Description

## Environment

- SDK: `github.com/lightning-ai/sdk/go`
- Version: `v0.0.0-20260903140713-8e137645d1a6`
- Upstream commit/tag: `8e137645d1a6b829533f0bf3cc11d2abc6c7291d` / `v2026.09.03`
- Machine: `CPU_SMALL` (`cpu-2`)
- Job: non-interruptible

The standalone-Job SDK contract says `0` means unset/backend default and `1`
means one attempt with no retries. Its request builder maps
`JobOptions.MaxRunAttempts` to `spec.maxRunAttempts`, and its tests verify that
field is serialized. The generated multi-machine schema is more specific: it
calls `0` "unset (legacy, no retries)." The corresponding guarantee is not
stated for standalone Jobs.

## Minimal reproduction

With `github.com/lightning-ai/sdk/go` imported as `lit` and authenticated
`teamspace` and `studio` handles already resolved:

```go
interruptible := false
job, err := lit.RunJob("retry-policy-reproducer", lit.Machine("cpu-2"), "true", lit.JobOptions{
Teamspace: teamspace,
Studio: studio,
Interruptible: &interruptible,
MaxRunAttempts: 1,
})
if err != nil {
log.Fatal(err)
}
defer func() { _ = job.Stop() }()

fmt.Printf("created: max=%d current=%d\n", job.MaxRunAttempts(), job.CurrentRunAttempt())

readback, err := lit.GetJob(job.Name(), lit.JobOptions{Teamspace: teamspace})
if err != nil {
log.Fatal(err)
}
fmt.Printf("readback: max=%d current=%d\n", readback.MaxRunAttempts(), readback.CurrentRunAttempt())
```

The observed job was submitted through the Python SDK with
`max_run_attempts=1`. Both Python and the pinned Go SDK read the same exact job
back as `0`; the Go SDK source and unit tests independently establish that its
create path serializes nonzero `MaxRunAttempts`. We did not spend another live
job merely to repeat the same backend request through Go.

## Expected

The created and subsequently fetched job retain `maxRunAttempts=1`, or the API
returns another authoritative field proving that the job cannot retry.

## Actual

The submission path set `max_run_attempts=1`, but the created job and later
exact-ID readback reported:

```text
current_run_attempt=1
max_run_attempts=0
```

The job was stopped before entering `Running`; `started_at` remained null.

## Impact

For a cost-bounded executor, `current_run_attempt=1` only describes the current
history. It does not prove that a second attempt cannot occur after a later
failure. Because the documented meaning of `0` is "unset/backend default," the
readback cannot attest the requested no-retry policy before compute effects.
The safe response is therefore to reject and stop the job.

## Questions

1. Is normalizing `maxRunAttempts=1` to `0` intentional?
2. For standalone Jobs, does `0` have the same stable legacy-no-retry meaning
documented for multi-machine jobs, or may a backend default retry?
3. If `0` guarantees no retries, can that guarantee be documented in the
standalone-Job API and exposed as an effective retry policy?
4. Otherwise, can the service preserve and return the explicit value `1`?

## References

- [SDK retry implementation PR](https://github.com/Lightning-AI/sdk/pull/142)
- [Pinned Go Job API](https://github.com/Lightning-AI/sdk/blob/v2026.09.03/go/job.go)
- [Latest SDK release](https://github.com/Lightning-AI/sdk/releases/tag/v2026.09.03)

Contributor guide

Open the contributing guide

Research direction

Start with the pinned Go Job API in go/job.go and the request-builder tests that verify MaxRunAttempts serialization. Compare the create request with the exact-ID readback and the generated multi-machine schema's treatment of zero. Done means establishing whether standalone Jobs preserve an explicit value of 1 or expose an authoritative no-retry policy, with the contract reflected in tests or documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, python
Domain
api, backend-api-design
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.