Azure / Azure/azure-sdk-for-python

Bug: Windows file-backed replay stream cannot recover after abrupt process exit

Open
#48,603 2 comments 1 reaction 1 assignee Claimed by @Shivakishore14 View on GitHub
Client customer-reported Hosted Agents needs-team-attention question Service Attention
Dominant language
Python
Stars
5.6k
Forks
3.4k
Avg merge
1d 21h
Merged PRs (30d)
193

Description

- **Package Name**: `azure-ai-agentserver-core`, `azure-ai-agentserver-responses`
- **Package Version**: `2.1.0b1`
- **Operating System**: Windows 11 Pro 10.0.26200 (build 26200)
- **Python Version**: 3.12.9, 64-bit

## Describe the bug

On Windows, `FileBackedReplayEventStream` uses an `O_CREAT | O_EXCL` `.lock` marker file as its single-writer lock. An abrupt process death cannot run `_cleanup_locks()`, so the marker remains on disk. A replacement Agent Server process then treats the stale marker as a live writer and cannot reopen the resilient stream. This blocks testing the resilient LRA feature locally on Windows dev machine.

This breaks Path-C recovery for stored background streaming responses. The response/task state can be recovered successfully, but `GET /responses/{id}?stream=true` repeatedly fails because the resilient stream cannot be opened.

This is Windows-specific:

- POSIX uses kernel-managed `fcntl.flock`; the kernel releases the lock when the process dies.
- Windows uses a persistent marker file; process death does not remove it.

The official Responses `CrashHarness` and resilience conformance suite are POSIX-only, so equivalent Windows crash recovery is currently not covered.

## Minimal reproduction

Run process A on Windows:

```python
import os
from pathlib import Path

from azure.ai.agentserver.core.streaming import FileBackedReplayEventStream

path = Path("repro-stream.jsonl")
stream = FileBackedReplayEventStream(path=path, cursor_fn=lambda event: event["n"])
os._exit(86)
```

Then run process B in the same directory:

```python
from pathlib import Path

from azure.ai.agentserver.core.streaming import FileBackedReplayEventStream

path = Path("repro-stream.jsonl")
stream = FileBackedReplayEventStream(path=path, cursor_fn=lambda event: event["n"])
```

Process B fails with:

```text
FileExistsError: [Errno 17] File exists: 'repro-stream.jsonl.lock'

RuntimeError: FileBackedReplayEventStream: another process holds the lock-file on repro-stream.jsonl
```

## Responses recovery reproduction

1. Start a `ResponsesAgentServerHost` on Windows with:
- `store=true`
- `background=true`
- `stream=true`
- `resilient_background=True`
- file-backed Agent Server state
2. Submit a response with a client-selected `x-agent-response-id`.
3. Abruptly terminate the process during the handler.
4. Restart with the same `AGENTSERVER_STATE_ROOT`.
5. Reconnect with `GET /responses/{id}?stream=true`.

Observed replacement-process log:

```text
WARNING azure.ai.agentserver.streaming: FileBackedReplayEventStream: lock-file contention on ...\streams\caresp_....jsonl

FileExistsError: [Errno 17] File exists: '...\streams\caresp_....jsonl.lock'

RuntimeError: FileBackedReplayEventStream: another process holds the lock-file on ...\streams\caresp_....jsonl
```

The same response can be visible through non-streaming retrieval as `in_progress`, while every streaming reconnect fails on the stale marker.

## Expected behavior

After the previous process is definitively dead, a replacement process should be able to reclaim and rehydrate the file-backed replay stream without manual `.lock` deletion.

Abrupt process loss is a supported Path-C condition. Recovery should not depend on graceful Python cleanup.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.