kubeflow / kubeflow/trainer

Trainer registers cluster-wide ConfigMap and Secret causing excessive memory consumption

Open
#3,374 5 comments 0 reactions 1 assignee Claimed by @beep-boopp View on GitHub
kind/bug
Dominant language
Go
Stars
2.2k
Forks
1.1k
Avg merge
3d 22h
Merged PRs (30d)
39

Description

### What happened?

The MPI plugin's ReconcilerBuilders() in pkg/runtime/framework/plugins/mpi/mpi.go registers cluster-wide watches on ConfigMap and Secret with no
label selector or namespace filter. While EnqueueRequestForOwner filters reconcile events to owner-referenced objects, the underlying informer
cache still lists and watches all ConfigMaps and Secrets cluster-wide, causing excessive memory consumption on clusters with many ConfigMaps.

Reproducer:
1. Install trainer v2.2.0 on KinD
2. Create 700 large ConfigMaps (200KB each)
3. Add a memory limit for Trainer operator and observe OOMKill

The controller OOMKills within seconds and restarts repeatedly. Without a memory limit, memory consumption reaches 415Mi with 700×200KB ConfigMaps in a fresh KinD cluster.

### What did you expect to happen?

The controller memory consumption should not grow with the number of unrelated ConfigMaps and Secrets in the cluster.

### Environment

Kubernetes version: v1.32.2 (KinD)
Kubeflow Trainer version: v2.2.0

### Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.