Runs lost if `fetch:credential` fails
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 21
- Forks
- 23
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 17
Description
Seeing a case in prod where credentials fail to load, for some reason
Logs like this:
[SRV] ✘ 3355001b-3ed7-4830-b6e7-42312b77f405 :: fetch:credential :: ERR: [fetch:credential] timeout
[SRV] ✘ 3355001b-3ed7-4830-b6e7-42312b77f405 :: fetch:credential :: ERR: [fetch:credential] timeout
[SRV] ✘ 3355001b-3ed7-4830-b6e7-42312b77f405 :: fetch:credential :: ERR: [fetch:credential] timeout
[SRV] ✘ 3355001b-3ed7-4830-b6e7-42312b77f405 :: fetch:credential :: ERR: [fetch:credential] timeout
[SRV] ✘ 3355001b-3ed7-4830-b6e7-42312b77f405 :: fetch:credential :: ERR: [fetch:credential] timeout
[SRV] ✘ 3355001b-3ed7-4830-b6e7-42312b77f405 :: run:log :: ERR: [run:log] timeout
INFO 2025-12-15T19:40:01.328077973Z [resource.labels.containerName: global-web] [warning] Detected lost run with reason LostAfterClaim:
[SRV] ❯ Critical error in channel run:3355001b-3ed7-4830-b6e7-42312b77f405 [
INFO 2025-12-16T00:42:14.683553324Z [resource.labels.containerName: global-web] [info] REFUSED JOIN run:3355001b-3ed7-4830-b6e7-42312b77f405 in 7ms
I think the lost is probably caused by a low grace period on the plan. But if the credential fails to load, really this run should return with an exception. So why didn't it?
Example run: GCP
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the worker's fetch:credential and run:log entry points using the reported logs and LostAfterClaim detection. Determine why a credential timeout does not return an exception before the run is lost, then add or update coverage for that failure path so the run terminates with the expected exception instead of being marked lost.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100