Lightning-AI / Lightning-AI/sdk
bug(jobs): maxRunAttempts=1 reads back as 0
- Dominant language
- Python
- Stars
- 10
- Forks
- 3
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 32
Description
## Environment
- SDK: `github.com/lightning-ai/sdk/go`
- Version: `v0.0.0-20260903140713-8e137645d1a6`
- Upstream commit/tag: `8e137645d1a6b829533f0bf3cc11d2abc6c7291d` / `v2026.09.03`
- Machine: `CPU_SMALL` (`cpu-2`)
- Job: non-interruptible
The standalone-Job SDK contract says `0` means unset/backend default and `1`
means one attempt with no retries. Its request builder maps
`JobOptions.MaxRunAttempts` to `spec.maxRunAttempts`, and its tests verify that
field is serialized. The generated multi-machine schema is more specific: it
calls `0` "unset (legacy, no retries)." The corresponding guarantee is not
stated for standalone Jobs.
## Minimal reproduction
With `github.com/lightning-ai/sdk/go` imported as `lit` and authenticated
`teamspace` and `studio` handles already resolved:
```go
interruptible := false
job, err := lit.RunJob("retry-policy-reproducer", lit.Machine("cpu-2"), "true", lit.JobOptions{
Teamspace: teamspace,
Studio: studio,
Interruptible: &interruptible,
MaxRunAttempts: 1,
})
if err != nil {
log.Fatal(err)
}
defer func() { _ = job.Stop() }()
fmt.Printf("created: max=%d current=%d\n", job.MaxRunAttempts(), job.CurrentRunAttempt())
readback, err := lit.GetJob(job.Name(), lit.JobOptions{Teamspace: teamspace})
if err != nil {
log.Fatal(err)
}
fmt.Printf("readback: max=%d current=%d\n", readback.MaxRunAttempts(), readback.CurrentRunAttempt())
```
The observed job was submitted through the Python SDK with
`max_run_attempts=1`. Both Python and the pinned Go SDK read the same exact job
back as `0`; the Go SDK source and unit tests independently establish that its
create path serializes nonzero `MaxRunAttempts`. We did not spend another live
job merely to repeat the same backend request through Go.
## Expected
The created and subsequently fetched job retain `maxRunAttempts=1`, or the API
returns another authoritative field proving that the job cannot retry.
## Actual
The submission path set `max_run_attempts=1`, but the created job and later
exact-ID readback reported:
```text
current_run_attempt=1
max_run_attempts=0
```
The job was stopped before entering `Running`; `started_at` remained null.
## Impact
For a cost-bounded executor, `current_run_attempt=1` only describes the current
history. It does not prove that a second attempt cannot occur after a later
failure. Because the documented meaning of `0` is "unset/backend default," the
readback cannot attest the requested no-retry policy before compute effects.
The safe response is therefore to reject and stop the job.
## Questions
1. Is normalizing `maxRunAttempts=1` to `0` intentional?
2. For standalone Jobs, does `0` have the same stable legacy-no-retry meaning
documented for multi-machine jobs, or may a backend default retry?
3. If `0` guarantees no retries, can that guarantee be documented in the
standalone-Job API and exposed as an effective retry policy?
4. Otherwise, can the service preserve and return the explicit value `1`?
## References
- [SDK retry implementation PR](https://github.com/Lightning-AI/sdk/pull/142)
- [Pinned Go Job API](https://github.com/Lightning-AI/sdk/blob/v2026.09.03/go/job.go)
- [Latest SDK release](https://github.com/Lightning-AI/sdk/releases/tag/v2026.09.03)
Contributor guide
Research direction
Start with the pinned Go Job API in go/job.go and the request-builder tests that verify MaxRunAttempts serialization. Compare the create request with the exact-ID readback and the generated multi-machine schema's treatment of zero. Done means establishing whether standalone Jobs preserve an explicit value of 1 or expose an authoritative no-retry policy, with the contract reflected in tests or documentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, python
- Domain
- api, backend-api-design
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100