tensorflow / tensorflow/tensorflow

Kubernetes cluster resolver fails when running from within a K8S cluster.

Open
#70,581 1 comment 0 reactions 1 assignee View on GitHub

@Kayyuri is already working on this.

Since Jun 16, 2026.

comp:dist-strat stat:contribution welcome TF 2.16 type:bug
Dominant language
C++
Stars
200k
Forks
76.9k
Avg merge
2d 3h
Merged PRs (30d)
433

Description

Issue type

Bug

Have you reproduced the bug with TensorFlow Nightly?

No

Source

binary

TensorFlow version

2.16.1

Custom code

Yes

OS platform and distribution

linux

Mobile device

No response

Python version

3.11

Bazel version

No response

GCC/compiler version

No response

CUDA/cuDNN version

No response

GPU model and memory

No response

Current behavior?

If trying to create a cluster spec from a pod running within a K8s cluster, this try block fails because it can't find the kubectl config file.

The quick is rather straightforward:


if override_client is None:
    try:
      from kubernetes import config as k8sconfig  # pylint: disable=g-import-not-at-top

      k8sconfig.load_kube_config()
    except ImportError:
      if not override_client:
        raise ImportError('The Kubernetes Python client must be installed '
                          'before using the Kubernetes Cluster Resolver. '
                          'To install the Kubernetes Python client, run '
                          '`pip install kubernetes` on your command line.')

...

Happy to open a MR for this.

Standalone code to reproduce the issue

main.py

import os
import tensorflow as tf

from absl import logging
from kubernetes import client, config

logging.set_verbosity(logging.DEBUG)
logging.info("TF version: %s", tf.__version__)

config.load_incluster_config()
k8s_cli = client.CoreV1Api()

# Fails here despite providing an override client for talking with the k8s APIs.
cluster_resolver = tf.distribute.cluster_resolver.KubernetesClusterResolver(
    {"worker": ["job-name=mobileye-0", "job-name=mobileye-1"]}, override_client=k8s_cli
)
task_index = int(os.environ.get("TASK_INDEX"))
cluster_resolver.task_type = "worker"
cluster_resolver.task_id = task_index

logging.info("Cluster spec: %s", cluster_resolver.cluster_spec().as_dict())
strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy(
    cluster_resolver=cluster_resolver
)

job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: mobileye-0
spec:
  template:
    metadata:
      name: mobileye-training
    spec:
      containers:
      - name: tensorflow
        image: europe-west4-docker.pkg.dev/msteiner-kubeflow/mobileye-test/test-tf-image:latest 
        resources:
          limits:
            cpu: "1"
            memory: 3Gi
        env:
          - name: TASK_INDEX
            value: "0"
      restartPolicy: Never
  parallelism: 1
---
apiVersion: batch/v1
kind: Job
metadata:
  name: mobileye-1
spec:
  template:
    metadata:
      name: mobileye-training
    spec:
      containers:
      - name: tensorflow
        image: europe-west4-docker.pkg.dev/msteiner-kubeflow/mobileye-test/test-tf-image:latest 
        resources:
          limits:
            cpu: "1"
            memory: 3Gi
        env:
          - name: TASK_INDEX
            value: "1"
      restartPolicy: Never
  parallelism: 1



### Relevant log output

```shell
2024-06-28 13:23:10.980 CEST
2024-06-28 11:23:10.979832: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-06-28 13:23:10.991 CEST
2024-06-28 11:23:10.991166: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-06-28 13:23:11.109 CEST
2024-06-28 11:23:11.108950: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
2024-06-28 13:23:11.109 CEST
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-06-28 13:23:14.057 CEST
2024-06-28 11:23:14.056964: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-06-28 13:23:18.311 CEST
INFO:absl:TF version: 2.16.1
2024-06-28 13:23:18.312 CEST
INFO:absl:PATH: /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-06-28 13:23:18.312 CEST
INFO:absl:HOSTNAME: mobileye-0-v7lkr
2024-06-28 13:23:18.312 CEST
INFO:absl:LANG: C.UTF-8
2024-06-28 13:23:18.312 CEST
INFO:absl:GPG_KEY: A035C8C19219BA821ECEA86B64E628F8D684696D
2024-06-28 13:23:18.312 CEST
INFO:absl:PYTHON_VERSION: 3.11.8
2024-06-28 13:23:18.312 CEST
INFO:absl:PYTHON_PIP_VERSION: 24.0
2024-06-28 13:23:18.312 CEST
INFO:absl:PYTHON_SETUPTOOLS_VERSION: 65.5.1
2024-06-28 13:23:18.312 CEST
INFO:absl:PYTHON_GET_PIP_URL: https://github.com/pypa/get-pip/raw/dbf0c85f76fb6e1ab42aa672ffca6f0a675d9ee4/public/get-pip.py
2024-06-28 13:23:18.312 CEST
INFO:absl:PYTHON_GET_PIP_SHA256: dfe9fd5c28dc98b5ac17979a953ea550cec37ae1b47a5116007395bfacff2ab9
2024-06-28 13:23:18.312 CEST
INFO:absl:TASK_INDEX: 0
2024-06-28 13:23:18.313 CEST
INFO:absl:KUBERNETES_SERVICE_PORT: 443
2024-06-28 13:23:18.313 CEST
INFO:absl:KUBERNETES_SERVICE_PORT_HTTPS: 443
2024-06-28 13:23:18.313 CEST
INFO:absl:KUBERNETES_PORT: tcp://34.118.224.1:443
2024-06-28 13:23:18.313 CEST
INFO:absl:KUBERNETES_PORT_443_TCP: tcp://34.118.224.1:443
2024-06-28 13:23:18.313 CEST
INFO:absl:KUBERNETES_PORT_443_TCP_PROTO: tcp
2024-06-28 13:23:18.313 CEST
INFO:absl:KUBERNETES_PORT_443_TCP_PORT: 443
2024-06-28 13:23:18.313 CEST
INFO:absl:KUBERNETES_PORT_443_TCP_ADDR: 34.118.224.1
2024-06-28 13:23:18.313 CEST
INFO:absl:KUBERNETES_SERVICE_HOST: 34.118.224.1
2024-06-28 13:23:18.313 CEST
INFO:absl:HOME: /root
2024-06-28 13:23:18.313 CEST
INFO:absl:TF2_BEHAVIOR: 1
2024-06-28 13:23:18.313 CEST
INFO:absl:TPU_ML_PLATFORM: Tensorflow
2024-06-28 13:23:18.315 CEST
Traceback (most recent call last):   File "//src/main.py", line 20, in <module>     cluster_resolver = tf.distribute.cluster_resolver.KubernetesClusterResolver(
2024-06-28 13:23:18.316 CEST
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-28 13:23:18.316 CEST
  File "/usr/local/lib/python3.11/site-packages/tensorflow/python/distribute/cluster_resolver/kubernetes_cluster_resolver.py", line 93, in __init__
2024-06-28 13:23:18.317 CEST
    k8sconfig.load_kube_config()
2024-06-28 13:23:18.317 CEST
  File "/usr/local/lib/python3.11/site-packages/kubernetes/config/kube_config.py", line 819, in load_kube_config
2024-06-28 13:23:18.318 CEST
    loader = _get_kube_config_loader(
2024-06-28 13:23:18.318 CEST
             ^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-28 13:23:18.318 CEST
  File "/usr/local/lib/python3.11/site-packages/kubernetes/config/kube_config.py", line 776, in _get_kube_config_loader
2024-06-28 13:23:18.319 CEST
    raise ConfigException(
2024-06-28 13:23:18.319 CEST
kubernetes.config.config_exception.ConfigException: Invalid kube-config file. No configuration found.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.