NVIDIA / NVIDIA/cluster-readiness-engine

fix(controller): prevent false WorkloadRun failure on Workflow cache miss

Open
#352 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
55
Forks
19
Avg merge
1d 5h
Merged PRs (30d)
107

Description

Problem

After a WorkloadRun creates its Workflow and persists status.workflowRef, a subsequent reconcile can observe the updated WorkloadRun before the controller cache observes the new Workflow. mirrorWorkflowStatus treats the cached NotFound as a real deletion and persists terminal WorkloadRunFailed / WorkflowDeleted.

Once terminal, the WorkloadRun never mirrors the Workflow's later result. ADR-080 makes the pre-existing condition user-visible as a misleading Warning / WorkflowDeleted Event.

This was reproduced while reviewing #349: workloadrun-event-success failed under race-instrumented timing immediately after Normal / WorkflowCreated. The Go race detector is not required for the bug; it only widens the informer-ordering window. Normal binaries can encounter the same sequence under cache latency or load.

Expected behavior

A cache miss immediately after Workflow creation must not be treated as proof that the Workflow was deleted. The reconciler should distinguish informer lag from confirmed absence and retry without writing a terminal failure.

Scope

  • Preserve the existing behavior for a Workflow confirmed to have been deleted.
  • Do not emit WorkflowDeleted or set WorkloadRun Failed for a transient cache miss.
  • Add deterministic coverage for both cache lag and confirmed deletion.
  • Keep the lifecycle/event semantics aligned with ADR-080.

Follow-up from #349; the PR's success fixture is hardened separately so this behavior change can be designed and reviewed on its own.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in the controller's mirrorWorkflowStatus reconciliation path and trace how a cached NotFound is handled after status.workflowRef is persisted. Use the workloadrun-event-success scenario from #349 as the reproduction context, then add deterministic coverage for cache lag and confirmed deletion. Done means cache lag retries without WorkflowDeleted or WorkloadRunFailed, while confirmed deletion preserves the existing behavior and ADR-080 event semantics.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.