InseeFrLab / InseeFrLab/auto-tuning-vllm

CLI hangs after trial completes when launched from K8s UI terminal and session is closed

Open
#44 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
6
Forks
0
Avg merge
34m
Merged PRs (30d)
2

Description

## Summary

When `auto-tune-vllm optimize` is started from an interactive Kubernetes UI terminal (exec) and the user closes the tab / disconnects, the optimization can appear to hang after a trial finishes:

- The trial worker completes successfully (benchmark + vLLM cleanup OK in trial logs)
- The main `auto-tune-vllm` process stays alive but stops making progress
- Optuna is not updated (`study.tell()` never runs or the main loop never continues)
- No subsequent trial is submitted

The same study configuration works reliably when:
- The user stays connected to the terminal, or
- The command is run as a Kubernetes Job (non-interactive stdout)

This was initially misdiagnosed as SQLite/NFS locking or trace-replay-specific behaviour, but reproduces across benchmark types when the terminal session is detached.

## Root cause

The CLI uses Rich live UI (`Progress` spinner + `RichHandler`) writing continuously to stdout.

When the K8s web terminal session ends:
- stdout becomes a broken pipe or an unread pipe
- The main thread blocks on the next console write (Rich spinner refresh or `logger.info()` via RichHandler)
- The worker thread (trial execution in `ThreadPoolExecutor`) is independent and can finish normally
- The orchestration loop in `StudyController` never collects the result and never calls `study.tell()`

Relevant code: `auto_tune_vllm/cli/main.py` — `Progress(SpinnerColumn(), ...)` + `RichHandler`.

## Workarounds (today)

- Run as a Kubernetes Job instead of a detached exec session
- Launch with `nohup` and redirect stdout/stderr to a file should work as well

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in auto_tune_vllm/cli/main.py, reading the Rich Progress spinner and RichHandler setup and how the CLI starts StudyController. Reproduce the command with stdout detached or closed, then trace whether console writes block the orchestration loop. Done means a detached session still lets the completed trial be collected, study.tell() run, and the next trial start.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
cli, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.