kubeflow / kubeflow/trainer

Fix Kubernetes 1.37 integration tests without changing the Trainer API

Open
#4,006 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.2k
Forks
1.1k
Avg merge
3d 22h
Merged PRs (30d)
39

Description

## Summary

The Kubernetes 1.37 update in [PR #4005](https://github.com/kubeflow/trainer/pull/4005) updates the envtest version and generated Kubernetes API artifacts, but the Go integration tests currently fail during envtest CRD installation.

## Reproduction

```bash
make test-integration
```

The Makefile correctly selects Kubernetes 1.37.0 through `setup-envtest`. The API server starts, but CRD installation fails before any controller or webhook specs run.

## Current failures

Kubernetes rejects the existing CRD schemas with errors including:

- `x-kubernetes-validations estimated rule cost exceeds budget`
- CEL rules referencing an undefined `metadata.namespace` field
- Failures occur for Trainer CRDs and the external JobSet CRD

This is an envtest/Kubernetes 1.37 CRD validation compatibility issue, rather than a failure in the integration test assertions.

## Desired outcome

Run the controller and webhook integration tests against Kubernetes 1.37 without changing the public Trainer API or weakening production CRD validation unintentionally.

## Options to evaluate

1. **Fix the CEL rules in the API schema** (preferred long-term solution if API/schema changes are acceptable): simplify expensive expressions and add bounds such as `maxItems`, `maxProperties`, and `maxLength`; correct rules that reference fields absent from the schema.
2. **Use test-only CRD copies**: copy the CRDs to a temporary directory for envtest and remove/simplify only the problematic CEL rules. This preserves production manifests but does not exercise CEL validation in the controller integration suite.
3. **Split validation coverage**: use sanitized CRDs for controller/webhook integration tests and add separate tests for CEL/schema validation.
4. **Use an existing Kubernetes 1.37 cluster** if it accepts the CRDs, though this may not avoid the same validation errors.
5. **Evaluate disabling `CustomResourceValidationExpressions` in envtest** only as a temporary workaround; this would make the test environment unlike a normal Kubernetes 1.37 cluster and may not address all schema issues.

Using an older envtest Kubernetes version is not considered a solution because this update specifically targets Kubernetes 1.37 compatibility, and the current schemas also fail on several tested older versions.

## Acceptance criteria

- `make test-integration` passes with Kubernetes 1.37 assets, or the chosen test-only workaround is documented and reliable.
- No unintended changes to the public Trainer API.
- Production CRD validation remains covered by an appropriate test path.
- The fix is compatible with the Kubernetes 1.37 update from [PR #4005](https://github.com/kubeflow/trainer/pull/4005).

Contributor guide

Open the contributing guide

Research direction

Start with the Makefile's test-integration target and reproduce the envtest CRD installation failure using Kubernetes 1.37. Inspect the Trainer and external JobSet CRD schemas and their CEL rules, then choose and document a workaround or schema fix that preserves production validation. Done means make test-integration passes, the public Trainer API is unchanged, and production CRD validation remains covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.