Fix Kubernetes 1.37 integration tests without changing the Trainer API
- Dominant language
- Go
- Stars
- 2.2k
- Forks
- 1.1k
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 39
Description
## Summary
The Kubernetes 1.37 update in [PR #4005](https://github.com/kubeflow/trainer/pull/4005) updates the envtest version and generated Kubernetes API artifacts, but the Go integration tests currently fail during envtest CRD installation.
## Reproduction
```bash
make test-integration
```
The Makefile correctly selects Kubernetes 1.37.0 through `setup-envtest`. The API server starts, but CRD installation fails before any controller or webhook specs run.
## Current failures
Kubernetes rejects the existing CRD schemas with errors including:
- `x-kubernetes-validations estimated rule cost exceeds budget`
- CEL rules referencing an undefined `metadata.namespace` field
- Failures occur for Trainer CRDs and the external JobSet CRD
This is an envtest/Kubernetes 1.37 CRD validation compatibility issue, rather than a failure in the integration test assertions.
## Desired outcome
Run the controller and webhook integration tests against Kubernetes 1.37 without changing the public Trainer API or weakening production CRD validation unintentionally.
## Options to evaluate
1. **Fix the CEL rules in the API schema** (preferred long-term solution if API/schema changes are acceptable): simplify expensive expressions and add bounds such as `maxItems`, `maxProperties`, and `maxLength`; correct rules that reference fields absent from the schema.
2. **Use test-only CRD copies**: copy the CRDs to a temporary directory for envtest and remove/simplify only the problematic CEL rules. This preserves production manifests but does not exercise CEL validation in the controller integration suite.
3. **Split validation coverage**: use sanitized CRDs for controller/webhook integration tests and add separate tests for CEL/schema validation.
4. **Use an existing Kubernetes 1.37 cluster** if it accepts the CRDs, though this may not avoid the same validation errors.
5. **Evaluate disabling `CustomResourceValidationExpressions` in envtest** only as a temporary workaround; this would make the test environment unlike a normal Kubernetes 1.37 cluster and may not address all schema issues.
Using an older envtest Kubernetes version is not considered a solution because this update specifically targets Kubernetes 1.37 compatibility, and the current schemas also fail on several tested older versions.
## Acceptance criteria
- `make test-integration` passes with Kubernetes 1.37 assets, or the chosen test-only workaround is documented and reliable.
- No unintended changes to the public Trainer API.
- Production CRD validation remains covered by an appropriate test path.
- The fix is compatible with the Kubernetes 1.37 update from [PR #4005](https://github.com/kubeflow/trainer/pull/4005).
Contributor guide
Research direction
Start with the Makefile's test-integration target and reproduce the envtest CRD installation failure using Kubernetes 1.37. Inspect the Trainer and external JobSet CRD schemas and their CEL rules, then choose and document a workaround or schema fix that preserves production validation. Done means make test-integration passes, the public Trainer API is unchanged, and production CRD validation remains covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- infrastructure, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100