agentscope-ai / agentscope-ai/agentscope
[Feature]: native MLflow integration for experiment tracking & logging
- 主要言語
- Python
- スター
- 31.6k
- フォーク
- 3.5k
- 平均マージ
- 1日 16時間
- マージ済み PR(30日)
- 103
説明
## Summary
Currently, AgentScope does not provide built-in support for **MLflow** as a logging/tracking backend. Users who rely on MLflow for experiment management must manually instrument their agents, which is error-prone and not integrated with AgentScope's lifecycle hooks.
This issue proposes adding **official first-class MLflow support** so that runs, metrics, parameters, artifacts, and agent traces can be reported to an MLflow tracking server automatically.
---
## Motivation
MLflow is one of the most widely adopted open-source platforms for ML experiment tracking. As AgentScope grows in production use cases, teams need a reliable way to:
- [ ] Track token usage, latency, and cost per run/agent
- [ ] Log agent parameters (model name, temperature, system prompt, etc.) as MLflow params
- [ ] Record custom metrics (task success rate, tool call count, turn count) as MLflow metrics
- [ ] Store conversation history and intermediate artifacts
- [ ] Compare runs across different agent configurations via the MLflow UI
- [ ] Integrate with existing ML pipelines that already use MLflow
---
## Proposed API / Usage Example
Ideally, enabling MLflow logging should require minimal code changes:
```python
import agentscope
from agentscope.logging import MLflowLogger
agentscope.init(
logger=MLflowLogger(
tracking_uri="http://localhost:5000",
experiment_name="my-agent-experiment",
)
)
# Run is started automatically; metrics are logged per message/turn
with agentscope.start_run(run_name="run-001"):
agent = MyAgent(name="assistant", model_config_name="gpt-4o")
agent(Msg("user", "Hello, plan my week."))
```
The logger should hook into AgentScope's existing event system to automatically capture:
```python
# Auto-logged (no user code needed):
mlflow.log_param("model", agent.model_config_name)
mlflow.log_param("agent_name", agent.name)
mlflow.log_metric("input_tokens", ..., step=turn)
mlflow.log_metric("output_tokens", ..., step=turn)
mlflow.log_metric("latency_ms", ..., step=turn)
# Optional manual logging:
mlflow.log_artifact("conversation.json")
```
---
## Expected Behavior
- [ ] A new `MLflowLogger` class (or logger plugin) under `agentscope/logging/`
- [ ] Auto-start and auto-end MLflow runs aligned with AgentScope run lifecycle
- [ ] Per-turn metric logging with `step` parameter
- [ ] Support for nested runs (for multi-agent pipelines)
- [ ] Graceful fallback if MLflow is not installed (`mlflow` as optional dependency)
- [ ] Documentation and a minimal working example in `examples/`
---
## Alternatives Considered
> Users can manually call `mlflow.log_metric()` inside agent logic, but this is not integrated with AgentScope's message/event lifecycle and requires duplicated boilerplate across projects. A similar pattern exists in LangChain (`MlflowCallbackHandler`) and LlamaIndex, which can serve as reference implementations.
---
## Additional Context
Related projects for reference:
- `langchain-community`: [`MlflowCallbackHandler`](https://python.langchain.com/docs/integrations/callbacks/mlflow_tracking)
- MLflow docs on LLM Tracing: https://mlflow.org/docs/latest/llms/tracing/index.html
- MLflow auto-logging for OpenAI: `mlflow.openai.autolog()`
コントリビューションガイド
評価
この issue はまだ評価されていません。