canonical / canonical/data-science-stack

Canonical k8s MLflow not working

Open
#193 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
37
Forks
10
PR merge metrics
No merged PRs in 30d

Description

### Bug Description

Details about the setup can be found in this comment https://github.com/canonical/data-science-stack/issues/187#issuecomment-2604444833

After deploying DSS to Canonical k8s MLflow is not working. The gui is reporting problem with

![Image](https://github.com/user-attachments/assets/905fcff8-0ede-4d04-9d04-6c8fe16b7e6a)

The browser console reports it has problems to list experiments

![Image](https://github.com/user-attachments/assets/3d512d21-2f18-426e-a1f3-04b3b1eeb500)

After deploying test notebook. Notebook is not able to write or search experiments in MLflow.

```
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
import mlflow
import mlflow.sklearn

# Generate a simple dataset
np.random.seed(42)
X = np.random.rand(100, 1) * 10 # Features
y = 2.5 * X + np.random.randn(100, 1) * 2 # Labels

# Split the dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Start an MLflow experiment
mlflow.set_experiment("Simple Linear Regression")

with mlflow.start_run():
# Train a linear regression model
model = LinearRegression()
model.fit(X_train, y_train)

# Make predictions and evaluate the model
y_pred = model.predict(X_test)
mse = mean_squared_error(y_test, y_pred)

# Log parameters, metrics, and the model
mlflow.log_param("fit_intercept", model.fit_intercept)
mlflow.log_metric("mse", mse)
mlflow.sklearn.log_model(model, "model")

print(f"Logged model with MSE: {mse}")

# Load the model back for inference
logged_model = mlflow.get_artifact_uri("model")
print(f"Model saved at: {logged_model}")
```

### To Reproduce

1. Run steps from this comment https://github.com/canonical/data-science-stack/issues/187#issuecomment-2604444833
2. Create a notebook `data-science-stack.dss create my-notebook-scipy --image=kubeflownotebookswg/jupyter-scipy:v1.8.0`
3. Install mlflow==2.1.1 inside the notebook.
4. Run the code from above
5. Run `dss status` to get mlflow url to access the UI

### Environment

Canonical k8s v1.32.0
DSS latest stable (without the microk8s storage class refference)

### Relevant Log Output

```shell
---------------------------------------------------------------------------
RestException Traceback (most recent call last)
Cell In[1], line 18
15 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
17 # Start an MLflow experiment
---> 18 mlflow.set_experiment("Simple Linear Regression")
20 with mlflow.start_run():
21 # Train a linear regression model
22 model = LinearRegression()

File /opt/conda/lib/python3.11/site-packages/mlflow/tracking/fluent.py:157, in set_experiment(experiment_name, experiment_id)
155 with _experiment_lock:
156 if experiment_id is None:
--> 157 experiment = client.get_experiment_by_name(experiment_name)
158 if not experiment:
159 try:

File /opt/conda/lib/python3.11/site-packages/mlflow/tracking/client.py:1257, in MlflowClient.get_experiment_by_name(self, name)
1225 def get_experiment_by_name(self, name: str) -> Optional[Experiment]:
1226 """Retrieve an experiment by experiment name from the backend store
1227
1228 Args:
(...)
1255 Lifecycle_stage: active
1256 """
-> 1257 return self._tracking_client.get_experiment_by_name(name)

File /opt/conda/lib/python3.11/site-packages/mlflow/tracking/_tracking_service/client.py:502, in TrackingServiceClient.get_experiment_by_name(self, name)
494 def get_experiment_by_name(self, name):
495 """
496 Args:
497 name: The experiment name.
(...)
500 :py:class:`mlflow.entities.Experiment`
501 """
--> 502 return self.store.get_experiment_by_name(name)

File /opt/conda/lib/python3.11/site-packages/mlflow/store/tracking/rest_store.py:522, in RestStore.get_experiment_by_name(self, experiment_name)
520 try:
521 req_body = message_to_json(GetExperimentByName(experiment_name=experiment_name))
--> 522 response_proto = self._call_endpoint(GetExperimentByName, req_body)
523 return Experiment.from_proto(response_proto.experiment)
524 except MlflowException as e:

File /opt/conda/lib/python3.11/site-packages/mlflow/store/tracking/rest_store.py:82, in RestStore._call_endpoint(self, api, json_body, endpoint)
80 endpoint, method = _METHOD_TO_INFO[api]
81 response_proto = api.Response()
---> 82 return call_endpoint(self.get_host_creds(), endpoint, method, json_body, response_proto)

File /opt/conda/lib/python3.11/site-packages/mlflow/utils/rest_utils.py:370, in call_endpoint(host_creds, endpoint, method, json_body, response_proto, extra_headers)
367 call_kwargs["json"] = json_body
368 response = http_request(**call_kwargs)
--> 370 response = verify_rest_response(response, endpoint)
371 response_to_parse = response.text
372 js_dict = json.loads(response_to_parse)

File /opt/conda/lib/python3.11/site-packages/mlflow/utils/rest_utils.py:240, in verify_rest_response(response, endpoint)
238 if response.status_code != 200:
239 if _can_parse_as_json_object(response.text):
--> 240 raise RestException(json.loads(response.text))
241 else:
242 base_msg = (
243 f"API request to endpoint {endpoint} "
244 f"failed with error code {response.status_code} != 200"
245 )

RestException: INVALID_PARAMETER_VALUE: Invalid experiment ID: 'lost+found'
```

### Additional Context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.