Azure / Azure/azure-sdk-for-python

azureml-mlflow ignores enable_async_logging

Open
#45,403 1 comment 0 reactions 0 assignees View on GitHub
azureml-mlflow customer-reported needs-team-attention question Service Attention
Dominant language
Python
Stars
5.6k
Forks
3.4k
Avg merge
1d 21h
Merged PRs (30d)
193

Description

- **Package Name**:
azureml-mlflow
- **Package Version**:
azureml-mlflow==1.60.0
mlflow==3.10.0

- **Operating System**:
Ubuntu

- **Python Version**:
3.12.12

**Describe the bug**
`mlflow.config.enable_async_logging(True)` appears to have no effect when using the `azureml-mlflow` tracking store (azureml://...). Calls to `mlflow.log_metrics()` block for ~700ms per call (and I observed delays up to several seconds during experimentation), instead of returning quickly and letting a background worker flush metrics asynchronously.
This behavior suggests the call is waiting on synchronous I/O (likely a service round-trip) instead of enqueueing metrics for background flushing.

**To Reproduce**
```
import time
import os
import mlflow
import mlflow.config
import mlflow.environment_variables

# Clear AzureML's run ID so we can start a fresh run
os.environ.pop("MLFLOW_RUN_ID", None)

N_LOGS = 20
step_offset = 0

# Enable async logging
mlflow.config.enable_async_logging(True)

# Start a run directly via the fluent API
mlflow.set_experiment("bench_mlflow_fluent")
mlflow.start_run()

print(f"MLflow tracking URI: {mlflow.get_tracking_uri()}")
print(f"Async logging enabled: {mlflow.environment_variables.MLFLOW_ENABLE_ASYNC_LOGGING.get()}")
print(f"Logging {N_LOGS} metric sets...\n")

times_ms = []
for i in range(N_LOGS):
step = step_offset + i

t0 = time.perf_counter()
mlflow.log_metrics({"bench/loss": 1.0 / (i + 1), "bench/lr": 1e-4 * (0.99 ** i)}, step=step)
elapsed_ms = (time.perf_counter() - t0) * 1000

times_ms.append(elapsed_ms)
print(f" step {step}: {elapsed_ms:.1f} ms")

print(f"\nSummary: min={min(times_ms):.1f}ms median={sorted(times_ms)[len(times_ms)//2]:.1f}ms max={max(times_ms):.1f}ms")

mlflow.end_run()
```
The output that I got when running interactively on an AzureML node:
```
MLflow tracking URI: azureml://westus2.api.azureml.ms/mlflow/v1.0/subscriptions/
Async logging enabled: True
Logging 20 metric sets...

step 0: 1054.3 ms
step 1: 717.3 ms
step 2: 768.9 ms
step 3: 739.6 ms
step 4: 725.1 ms
step 5: 719.2 ms
step 6: 734.3 ms
step 7: 749.1 ms
step 8: 713.6 ms
step 9: 763.8 ms
step 10: 708.6 ms
step 11: 724.5 ms
step 12: 723.2 ms
step 13: 715.7 ms
step 14: 741.0 ms
step 15: 714.0 ms
step 16: 717.7 ms
step 17: 748.0 ms
step 18: 725.3 ms
step 19: 728.1 ms

Summary: min=708.6ms median=725.3ms max=1054.3ms
```

**Expected behavior**
With async logging enabled, `mlflow.log_metrics()` should not block on network/service I/O. It should enqueue metrics and return quickly (e.g., within a few ms), with a background thread/process responsible for communicating with the tracking server.

**Screenshots**
See the output above.

**Additional context**
This affects training loops on AzureML that log metrics frequently. The cumulative per-call blocking time adds significant overhead.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.