NVIDIA / NVIDIA/nvcf

test(bdd): cover durable cordon-and-drain recovery

Open
#1,186 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

clone-disabled
Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 12h
Merged PRs (30d)
427

Description

Description

Add live smoke coverage for the self-managed cluster maintenance workflow. On a split local topology, nvcf-cli cluster agent cordon-and-drain can report success and complete the NVCA rollout, but subsequent operator reconciliation can remove the maintenance configuration. The function remains degraded, and cluster agent uncordon reports that no change is needed instead of restoring scheduling.

Cover the workflow in tests/bdd on the local multi-cluster topology, where the control plane and compute plane run in separate k3d clusters. Reuse the existing sample-function lifecycle and the strict DSL. Express maintenance operations and state checks with generic command and output assertions instead of adding domain-specific step handlers.

The smoke flow should prove that maintenance remains active across NVCA rollout and operator reconciliation, drains the function to zero instances, and permits an explicit uncordon to reschedule and restore the function.

Definition of Done

  • Add or extend a live scenario under tests/bdd/features/ for the split local control-plane and compute-plane topology.
  • Create and deploy the sample function, verify it reaches ACTIVE, and confirm it can be invoked before maintenance.
  • Run cluster agent cordon-and-drain non-interactively against the expected compute cluster and assert the command succeeds.
  • Wait for the NVCA rollout and operator reconciliation, then verify CordonAndDrain remains the effective maintenance mode and the function drains to zero instances.
  • Run cluster agent uncordon and verify it performs a state change instead of reporting that the cluster is already uncordoned.
  • Verify maintenance is removed, the function is rescheduled, returns to ACTIVE, and can be invoked again.
  • Use the existing strict BDD vocabulary. Do not add domain-specific step handlers for cluster maintenance.
  • Add fake-runner wiring coverage in tests/bdd/godog_test.go for the new scenario.
  • Run go test -short ./..., tests/bdd/scripts/lint.sh, and the affected live smoke scenario. Document any live test that cannot be run locally.

Resources (optional)

  • tests/bdd/AGENTS.md
  • tests/bdd/PLAN.md
  • tests/bdd/features/multi-cluster-helmfile.feature
  • src/clis/nvcf-cli/README.md

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tests/bdd/AGENTS.md and PLAN.md, then inspect tests/bdd/features/multi-cluster-helmfile.feature and tests/bdd/godog_test.go for the existing lifecycle, strict DSL, and fake-runner wiring. Run the short Go suite and BDD lint before exercising the live split-cluster scenario. Done means the scenario covers cordon-and-drain through rollout and reconciliation, zero instances, explicit uncordon, rescheduling, ACTIVE state, and invocation recovery.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
infrastructure, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.