kubeflow / kubeflow/notebooks

Directly support internal minio artifacts in tensorboard

Open
#733 6 comments 0 reactions 0 assignees View on GitHub
area/v1 kind/enhancement priority/needs-triage
Dominant language
No language data
Stars
84
Forks
149
Avg merge
5d 15h
Merged PRs (30d)
29

Description

### Checks

- [x] I have searched the [existing issues](https://github.com/kubeflow/notebooks/issues).
- [x] My request is related to one of the components in the [`kubeflow/notebooks`](https://github.com/kubeflow/notebooks) repository.

### Motivation

TL;DR: Currently its not possible to use a kfp artifact (in kubeflow's minio) as object storage target in a tensorboard server in an straightforward manner.

- Current behaviour:
1. Create a tensorboard server with object storage. with an uri like "minio://path/to/artifact"
2. Server creation will be stuck forever (No user facing error whatsoever)
3. Controller logs show that a `tb-volume` is missing (Really misleading)

This is because [we currently only handle gcs or s3 prefixes](https://github.com/kubeflow/notebooks/blob/8c44fd768b61bdf7c459cd3a248a31dee481edf2/components/tensorboard-controller/controllers/tensorboard_controller.go#L178C6-L178C17).

- Expected behaviour:
1. Create a tensorboard server with object storage. with an uri like "minio://path/to/artifact"
2. The controller handles the `minio://` prefix and defaults to use the internal minio service url and use the crendentials from mlpipeline-minio-artifact

- Current workaround (Still not tested)
1. Use `s3://` prefix in place of `minio://`
2. Use a podDefault setting the minio server url and credentials

### Implementation

_No response_

### Are you willing & able to help?

- [ ] I am able to submit a PR!
- [x] I can help test the feature!

Contributor guide

Open the contributing guide

Research direction

Start in components/tensorboard-controller/controllers/tensorboard_controller.go at the object-storage prefix handling referenced by the issue, then trace how TensorBoard server creation resolves storage and credentials. Done means a minio:// artifact uses the internal Minio service and mlpipeline-minio-artifact credentials without the tb-volume failure; verify the existing workaround and test the server creation flow.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes
Domain
backend, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.