NVIDIA / NVIDIA/cluster-readiness-engine
fix(controller): prevent false WorkloadRun failure on Workflow cache miss
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 55
- Forks
- 19
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 107
Description
Problem
After a WorkloadRun creates its Workflow and persists status.workflowRef, a subsequent reconcile can observe the updated WorkloadRun before the controller cache observes the new Workflow. mirrorWorkflowStatus treats the cached NotFound as a real deletion and persists terminal WorkloadRunFailed / WorkflowDeleted.
Once terminal, the WorkloadRun never mirrors the Workflow's later result. ADR-080 makes the pre-existing condition user-visible as a misleading Warning / WorkflowDeleted Event.
This was reproduced while reviewing #349: workloadrun-event-success failed under race-instrumented timing immediately after Normal / WorkflowCreated. The Go race detector is not required for the bug; it only widens the informer-ordering window. Normal binaries can encounter the same sequence under cache latency or load.
Expected behavior
A cache miss immediately after Workflow creation must not be treated as proof that the Workflow was deleted. The reconciler should distinguish informer lag from confirmed absence and retry without writing a terminal failure.
Scope
- Preserve the existing behavior for a Workflow confirmed to have been deleted.
- Do not emit
WorkflowDeletedor set WorkloadRunFailedfor a transient cache miss. - Add deterministic coverage for both cache lag and confirmed deletion.
- Keep the lifecycle/event semantics aligned with ADR-080.
Follow-up from #349; the PR's success fixture is hardened separately so this behavior change can be designed and reviewed on its own.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in the controller's mirrorWorkflowStatus reconciliation path and trace how a cached NotFound is handled after status.workflowRef is persisted. Use the workloadrun-event-success scenario from #349 as the reproduction context, then add deterministic coverage for cache lag and confirmed deletion. Done means cache lag retries without WorkflowDeleted or WorkloadRunFailed, while confirmed deletion preserves the existing behavior and ADR-080 event semantics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 64/100