[BUG] genai.evaluate does not work with mlflow + sagemaker-mlflow
- Dominant language
- Python
- Stars
- 26
- Forks
- 19
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
### MLflow version
- Client: 3.8.0
### System information
```
[project]
name = "prompts"
version = "0.1.0"
description = "Prompts package"
readme = "README.md"
authors = [
{ }
]
requires-python = "==3.10.*"
dependencies = [
"mlflow==3.8.0",
"PyYAML==6.0.2",
"pydantic-ai-slim[openai,bedrock,evals]>=1.8.0",
"litellm>=1.80.10",
"sagemaker-mlflow==0.2.0"
]
[project.scripts]
prompts = "prompts:main"
[build-system]
requires = ["uv_build>=0.9.15,<0.10.0"]
build-backend = "uv_build"
```
### Describe the problem
GenAI evaluation does not work with the sagemaker-mlflow. The same code runs fine when sagemaker-mlflow dependency ("sagemaker-mlflow==0.2.0") is removed from the uv pyproject.toml. Below is the error when sagemaker-mlflow is installed alongside mlflow.
### Tracking information
```shell
REPLACE_ME
```
### Code to reproduce issue
```
import mlflow
from mlflow.tracking import MlflowClient
from mlflow.genai import scorer
from mlflow.entities import Feedback
mlflow.set_tracking_uri("arn:aws:sagemaker:us-east-2:xxx")
client = MlflowClient()
mlflow.set_experiment('prompt-evaluation')
@scorer
def exact_match(outputs: dict, expectations: dict) -> bool:
return outputs == expectations["expected_response"]
@scorer
def is_short(outputs: dict) -> Feedback:
score = len(outputs.split()) <= 5
rationale = (
"The response is short enough."
if score
else f"The response is not short enough because it has ({len(outputs.split())} words)."
)
print('score:', score)
return Feedback(value=score, rationale=rationale)
eval_dataset = [
{
"inputs": {"question": "How many countries are there in the world?"},
"outputs": "195",
"expectations": {"expected_response": "195"},
},
{
"inputs": {"question": "What is the capital of France?"},
"outputs": "The capital of France is Paris.",
"expectations": {"expected_response": "Paris"},
},
]
mlflow.genai.evaluate(
data=eval_dataset,
scorers=[exact_match, is_short],
)
```
### Stack trace
```
2025/12/23 14:03:32 INFO mlflow.models.evaluation.utils.trace: Auto tracing is temporarily enabled during the model evaluation for computing some metrics and debugging. To disable tracing, call `mlflow.autolog(disable=True)`.
Evaluating: 0%| | 0/2 [Elapsed: 00:00, Remaining: ?] 2025/12/23 14:03:32 WARNING mlflow.tracing.fluent: Failed to get trace from the tracking store: API request to endpoint /api/2.0/mlflow/traces/tr-b4452c0f19c8e8004292877792b79ca8 failed with error code 403 != 200. Response body: 'Action string is null.' For full traceback, set logging level to debug.
2025/12/23 14:03:32 WARNING mlflow.tracing.fluent: Failed to get trace from the tracking store: API request to endpoint /api/2.0/mlflow/traces/tr-2469a7b84408888d8aac07fa90662e2a failed with error code 403 != 200. Response body: 'Action string is null.' For full traceback, set logging level to debug.
score: True
score: False
Evaluating: 0%| | 0/2 [Elapsed: 00:00, Remaining: ?]
Traceback (most recent call last):
File "/home/ml-platform/prompts/src/prompts/test_evaluate_prompt.py", line 39, in
mlflow.genai.evaluate(
File "/home/ml-platform/.venv/lib/python3.10/site-packages/mlflow/genai/evaluation/base.py", line 248, in evaluate
result, _ = _run_harness(data, scorers, predict_fn, model_id)
File "/home/ml-platform/.venv/lib/python3.10/site-packages/mlflow/telemetry/track.py", line 30, in wrapper
result = func(*args, **kwargs)
File "/home/ml-platform/.venv/lib/python3.10/site-packages/mlflow/genai/evaluation/base.py", line 328, in _run_harness
result = harness.run(
File "/home/ml-platform/.venv/lib/python3.10/site-packages/mlflow/genai/evaluation/context.py", line 137, in wrapper
return func(*args, **kwargs)
File "/home/ml-platform/.venv/lib/python3.10/site-packages/mlflow/genai/evaluation/harness.py", line 167, in run
eval_results[idx] = future.result()
File "/root/.local/share/uv/python/cpython-3.10.19-linux-x86_64-gnu/lib/python3.10/concurrent/futures/_base.py", line 451, in result
return self.__get_result()
File "/root/.local/share/uv/python/cpython-3.10.19-linux-x86_64-gnu/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/root/.local/share/uv/python/cpython-3.10.19-linux-x86_64-gnu/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/ml-platform/.venv/lib/python3.10/site-packages/mlflow/genai/evaluation/harness.py", line 275, in _run_single
assessments.extend(_get_new_expectations(eval_item))
File "/home/ml-platform/.venv/lib/python3.10/site-packages/mlflow/genai/evaluation/harness.py", line 379, in _get_new_expectations
a.name for a in eval_item.trace.info.assessments if a.expectation is not None
AttributeError: 'NoneType' object has no attribute 'info'
```
Any clarification is highly appreciated. Thank you.
Contributor guide
Research direction
Reproduce the failure with mlflow==3.8.0 and sagemaker-mlflow==0.2.0 using the shown mlflow.genai.evaluate entry point and sample dataset. Inspect the evaluation harness failure at _get_new_expectations and compare behavior with sagemaker-mlflow removed; done means the evaluation completes successfully with both dependencies installed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100