Azure / Azure/azure-sdk-for-python
Bug: Windows file-backed replay stream cannot recover after abrupt process exit
- Dominant language
- Python
- Stars
- 5.6k
- Forks
- 3.4k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 193
Description
- **Package Name**: `azure-ai-agentserver-core`, `azure-ai-agentserver-responses`
- **Package Version**: `2.1.0b1`
- **Operating System**: Windows 11 Pro 10.0.26200 (build 26200)
- **Python Version**: 3.12.9, 64-bit
## Describe the bug
On Windows, `FileBackedReplayEventStream` uses an `O_CREAT | O_EXCL` `.lock` marker file as its single-writer lock. An abrupt process death cannot run `_cleanup_locks()`, so the marker remains on disk. A replacement Agent Server process then treats the stale marker as a live writer and cannot reopen the resilient stream. This blocks testing the resilient LRA feature locally on Windows dev machine.
This breaks Path-C recovery for stored background streaming responses. The response/task state can be recovered successfully, but `GET /responses/{id}?stream=true` repeatedly fails because the resilient stream cannot be opened.
This is Windows-specific:
- POSIX uses kernel-managed `fcntl.flock`; the kernel releases the lock when the process dies.
- Windows uses a persistent marker file; process death does not remove it.
The official Responses `CrashHarness` and resilience conformance suite are POSIX-only, so equivalent Windows crash recovery is currently not covered.
## Minimal reproduction
Run process A on Windows:
```python
import os
from pathlib import Path
from azure.ai.agentserver.core.streaming import FileBackedReplayEventStream
path = Path("repro-stream.jsonl")
stream = FileBackedReplayEventStream(path=path, cursor_fn=lambda event: event["n"])
os._exit(86)
```
Then run process B in the same directory:
```python
from pathlib import Path
from azure.ai.agentserver.core.streaming import FileBackedReplayEventStream
path = Path("repro-stream.jsonl")
stream = FileBackedReplayEventStream(path=path, cursor_fn=lambda event: event["n"])
```
Process B fails with:
```text
FileExistsError: [Errno 17] File exists: 'repro-stream.jsonl.lock'
RuntimeError: FileBackedReplayEventStream: another process holds the lock-file on repro-stream.jsonl
```
## Responses recovery reproduction
1. Start a `ResponsesAgentServerHost` on Windows with:
- `store=true`
- `background=true`
- `stream=true`
- `resilient_background=True`
- file-backed Agent Server state
2. Submit a response with a client-selected `x-agent-response-id`.
3. Abruptly terminate the process during the handler.
4. Restart with the same `AGENTSERVER_STATE_ROOT`.
5. Reconnect with `GET /responses/{id}?stream=true`.
Observed replacement-process log:
```text
WARNING azure.ai.agentserver.streaming: FileBackedReplayEventStream: lock-file contention on ...\streams\caresp_....jsonl
FileExistsError: [Errno 17] File exists: '...\streams\caresp_....jsonl.lock'
RuntimeError: FileBackedReplayEventStream: another process holds the lock-file on ...\streams\caresp_....jsonl
```
The same response can be visible through non-streaming retrieval as `in_progress`, while every streaming reconnect fails on the stale marker.
## Expected behavior
After the previous process is definitively dead, a replacement process should be able to reclaim and rehydrate the file-backed replay stream without manual `.lock` deletion.
Abrupt process loss is a supported Path-C condition. Recovery should not depend on graceful Python cleanup.
Contributor guide
Assessment
This issue has not been assessed yet.