canonical / canonical/data-science-stack
Canonical k8s MLflow not working
- Dominant language
- Python
- Stars
- 37
- Forks
- 10
- PR merge metrics
- No merged PRs in 30d
Description
### Bug Description
Details about the setup can be found in this comment https://github.com/canonical/data-science-stack/issues/187#issuecomment-2604444833
After deploying DSS to Canonical k8s MLflow is not working. The gui is reporting problem with

The browser console reports it has problems to list experiments

After deploying test notebook. Notebook is not able to write or search experiments in MLflow.
```
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
import mlflow
import mlflow.sklearn
# Generate a simple dataset
np.random.seed(42)
X = np.random.rand(100, 1) * 10 # Features
y = 2.5 * X + np.random.randn(100, 1) * 2 # Labels
# Split the dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Start an MLflow experiment
mlflow.set_experiment("Simple Linear Regression")
with mlflow.start_run():
# Train a linear regression model
model = LinearRegression()
model.fit(X_train, y_train)
# Make predictions and evaluate the model
y_pred = model.predict(X_test)
mse = mean_squared_error(y_test, y_pred)
# Log parameters, metrics, and the model
mlflow.log_param("fit_intercept", model.fit_intercept)
mlflow.log_metric("mse", mse)
mlflow.sklearn.log_model(model, "model")
print(f"Logged model with MSE: {mse}")
# Load the model back for inference
logged_model = mlflow.get_artifact_uri("model")
print(f"Model saved at: {logged_model}")
```
### To Reproduce
1. Run steps from this comment https://github.com/canonical/data-science-stack/issues/187#issuecomment-2604444833
2. Create a notebook `data-science-stack.dss create my-notebook-scipy --image=kubeflownotebookswg/jupyter-scipy:v1.8.0`
3. Install mlflow==2.1.1 inside the notebook.
4. Run the code from above
5. Run `dss status` to get mlflow url to access the UI
### Environment
Canonical k8s v1.32.0
DSS latest stable (without the microk8s storage class refference)
### Relevant Log Output
```shell
---------------------------------------------------------------------------
RestException Traceback (most recent call last)
Cell In[1], line 18
15 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
17 # Start an MLflow experiment
---> 18 mlflow.set_experiment("Simple Linear Regression")
20 with mlflow.start_run():
21 # Train a linear regression model
22 model = LinearRegression()
File /opt/conda/lib/python3.11/site-packages/mlflow/tracking/fluent.py:157, in set_experiment(experiment_name, experiment_id)
155 with _experiment_lock:
156 if experiment_id is None:
--> 157 experiment = client.get_experiment_by_name(experiment_name)
158 if not experiment:
159 try:
File /opt/conda/lib/python3.11/site-packages/mlflow/tracking/client.py:1257, in MlflowClient.get_experiment_by_name(self, name)
1225 def get_experiment_by_name(self, name: str) -> Optional[Experiment]:
1226 """Retrieve an experiment by experiment name from the backend store
1227
1228 Args:
(...)
1255 Lifecycle_stage: active
1256 """
-> 1257 return self._tracking_client.get_experiment_by_name(name)
File /opt/conda/lib/python3.11/site-packages/mlflow/tracking/_tracking_service/client.py:502, in TrackingServiceClient.get_experiment_by_name(self, name)
494 def get_experiment_by_name(self, name):
495 """
496 Args:
497 name: The experiment name.
(...)
500 :py:class:`mlflow.entities.Experiment`
501 """
--> 502 return self.store.get_experiment_by_name(name)
File /opt/conda/lib/python3.11/site-packages/mlflow/store/tracking/rest_store.py:522, in RestStore.get_experiment_by_name(self, experiment_name)
520 try:
521 req_body = message_to_json(GetExperimentByName(experiment_name=experiment_name))
--> 522 response_proto = self._call_endpoint(GetExperimentByName, req_body)
523 return Experiment.from_proto(response_proto.experiment)
524 except MlflowException as e:
File /opt/conda/lib/python3.11/site-packages/mlflow/store/tracking/rest_store.py:82, in RestStore._call_endpoint(self, api, json_body, endpoint)
80 endpoint, method = _METHOD_TO_INFO[api]
81 response_proto = api.Response()
---> 82 return call_endpoint(self.get_host_creds(), endpoint, method, json_body, response_proto)
File /opt/conda/lib/python3.11/site-packages/mlflow/utils/rest_utils.py:370, in call_endpoint(host_creds, endpoint, method, json_body, response_proto, extra_headers)
367 call_kwargs["json"] = json_body
368 response = http_request(**call_kwargs)
--> 370 response = verify_rest_response(response, endpoint)
371 response_to_parse = response.text
372 js_dict = json.loads(response_to_parse)
File /opt/conda/lib/python3.11/site-packages/mlflow/utils/rest_utils.py:240, in verify_rest_response(response, endpoint)
238 if response.status_code != 200:
239 if _can_parse_as_json_object(response.text):
--> 240 raise RestException(json.loads(response.text))
241 else:
242 base_msg = (
243 f"API request to endpoint {endpoint} "
244 f"failed with error code {response.status_code} != 200"
245 )
RestException: INVALID_PARAMETER_VALUE: Invalid experiment ID: 'lost+found'
```
### Additional Context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.