agentscope-ai / agentscope-ai/agentscope

[Feature]: native MLflow integration for experiment tracking & logging

オープン
#1,458 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る
stale-issue
主要言語
Python
スター
31.6k
フォーク
3.5k
平均マージ
1日 16時間
マージ済み PR(30日)
103

説明

## Summary

Currently, AgentScope does not provide built-in support for **MLflow** as a logging/tracking backend. Users who rely on MLflow for experiment management must manually instrument their agents, which is error-prone and not integrated with AgentScope's lifecycle hooks.

This issue proposes adding **official first-class MLflow support** so that runs, metrics, parameters, artifacts, and agent traces can be reported to an MLflow tracking server automatically.

---

## Motivation

MLflow is one of the most widely adopted open-source platforms for ML experiment tracking. As AgentScope grows in production use cases, teams need a reliable way to:

- [ ] Track token usage, latency, and cost per run/agent
- [ ] Log agent parameters (model name, temperature, system prompt, etc.) as MLflow params
- [ ] Record custom metrics (task success rate, tool call count, turn count) as MLflow metrics
- [ ] Store conversation history and intermediate artifacts
- [ ] Compare runs across different agent configurations via the MLflow UI
- [ ] Integrate with existing ML pipelines that already use MLflow

---

## Proposed API / Usage Example

Ideally, enabling MLflow logging should require minimal code changes:

```python
import agentscope
from agentscope.logging import MLflowLogger

agentscope.init(
logger=MLflowLogger(
tracking_uri="http://localhost:5000",
experiment_name="my-agent-experiment",
)
)

# Run is started automatically; metrics are logged per message/turn
with agentscope.start_run(run_name="run-001"):
agent = MyAgent(name="assistant", model_config_name="gpt-4o")
agent(Msg("user", "Hello, plan my week."))
```

The logger should hook into AgentScope's existing event system to automatically capture:

```python
# Auto-logged (no user code needed):
mlflow.log_param("model", agent.model_config_name)
mlflow.log_param("agent_name", agent.name)
mlflow.log_metric("input_tokens", ..., step=turn)
mlflow.log_metric("output_tokens", ..., step=turn)
mlflow.log_metric("latency_ms", ..., step=turn)

# Optional manual logging:
mlflow.log_artifact("conversation.json")
```

---

## Expected Behavior

- [ ] A new `MLflowLogger` class (or logger plugin) under `agentscope/logging/`
- [ ] Auto-start and auto-end MLflow runs aligned with AgentScope run lifecycle
- [ ] Per-turn metric logging with `step` parameter
- [ ] Support for nested runs (for multi-agent pipelines)
- [ ] Graceful fallback if MLflow is not installed (`mlflow` as optional dependency)
- [ ] Documentation and a minimal working example in `examples/`

---

## Alternatives Considered

> Users can manually call `mlflow.log_metric()` inside agent logic, but this is not integrated with AgentScope's message/event lifecycle and requires duplicated boilerplate across projects. A similar pattern exists in LangChain (`MlflowCallbackHandler`) and LlamaIndex, which can serve as reference implementations.

---

## Additional Context

Related projects for reference:
- `langchain-community`: [`MlflowCallbackHandler`](https://python.langchain.com/docs/integrations/callbacks/mlflow_tracking)
- MLflow docs on LLM Tracing: https://mlflow.org/docs/latest/llms/tracing/index.html
- MLflow auto-logging for OpenAI: `mlflow.openai.autolog()`

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。