Azure / Azure/azure-sdk-for-python

azureml-mlflow ignores enable_async_logging

Aperta
#45,403 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
azureml-mlflow customer-reported needs-team-attention question Service Attention
Lingua principale
Python
Stelle
5.6k
Fork
3.4k
Merge medio
2g 2h
PR unite (30g)
213

Descrizione

- **Package Name**:
azureml-mlflow
- **Package Version**:
azureml-mlflow==1.60.0
mlflow==3.10.0

- **Operating System**:
Ubuntu

- **Python Version**:
3.12.12

**Describe the bug**
`mlflow.config.enable_async_logging(True)` appears to have no effect when using the `azureml-mlflow` tracking store (azureml://...). Calls to `mlflow.log_metrics()` block for ~700ms per call (and I observed delays up to several seconds during experimentation), instead of returning quickly and letting a background worker flush metrics asynchronously.
This behavior suggests the call is waiting on synchronous I/O (likely a service round-trip) instead of enqueueing metrics for background flushing.

**To Reproduce**
```
import time
import os
import mlflow
import mlflow.config
import mlflow.environment_variables

# Clear AzureML's run ID so we can start a fresh run
os.environ.pop("MLFLOW_RUN_ID", None)

N_LOGS = 20
step_offset = 0

# Enable async logging
mlflow.config.enable_async_logging(True)

# Start a run directly via the fluent API
mlflow.set_experiment("bench_mlflow_fluent")
mlflow.start_run()

print(f"MLflow tracking URI: {mlflow.get_tracking_uri()}")
print(f"Async logging enabled: {mlflow.environment_variables.MLFLOW_ENABLE_ASYNC_LOGGING.get()}")
print(f"Logging {N_LOGS} metric sets...\n")

times_ms = []
for i in range(N_LOGS):
step = step_offset + i

t0 = time.perf_counter()
mlflow.log_metrics({"bench/loss": 1.0 / (i + 1), "bench/lr": 1e-4 * (0.99 ** i)}, step=step)
elapsed_ms = (time.perf_counter() - t0) * 1000

times_ms.append(elapsed_ms)
print(f" step {step}: {elapsed_ms:.1f} ms")

print(f"\nSummary: min={min(times_ms):.1f}ms median={sorted(times_ms)[len(times_ms)//2]:.1f}ms max={max(times_ms):.1f}ms")

mlflow.end_run()
```
The output that I got when running interactively on an AzureML node:
```
MLflow tracking URI: azureml://westus2.api.azureml.ms/mlflow/v1.0/subscriptions/
Async logging enabled: True
Logging 20 metric sets...

step 0: 1054.3 ms
step 1: 717.3 ms
step 2: 768.9 ms
step 3: 739.6 ms
step 4: 725.1 ms
step 5: 719.2 ms
step 6: 734.3 ms
step 7: 749.1 ms
step 8: 713.6 ms
step 9: 763.8 ms
step 10: 708.6 ms
step 11: 724.5 ms
step 12: 723.2 ms
step 13: 715.7 ms
step 14: 741.0 ms
step 15: 714.0 ms
step 16: 717.7 ms
step 17: 748.0 ms
step 18: 725.3 ms
step 19: 728.1 ms

Summary: min=708.6ms median=725.3ms max=1054.3ms
```

**Expected behavior**
With async logging enabled, `mlflow.log_metrics()` should not block on network/service I/O. It should enqueue metrics and return quickly (e.g., within a few ms), with a background thread/process responsible for communicating with the tracking server.

**Screenshots**
See the output above.

**Additional context**
This affects training loops on AzureML that log metrics frequently. The cumulative per-call blocking time adds significant overhead.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia riproducendo il benchmark segnalato con la tracking URI azureml:// e verifica come il tracking store azureml-mlflow gestisce mlflow.config.enable_async_logging durante le chiamate a log_metrics. Confronta il percorso della chiamata con il comportamento asincrono previsto da MLflow. Il lavoro è completato quando il logging asincrono abilitato restituisce rapidamente senza attendere l’I/O del servizio, mentre le metriche vengono comunque scaricate correttamente.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
azure, python
Ambito
backend, cloud, performance
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Tranquilla
Chiarezza
Abbastanza chiara
Idoneità per principianti
45/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.