ROS3/S3 VFD deadlocks at process exit on Windows
- Dominant language
- C
- Stars
- 988
- Forks
- 355
- Avg merge
- 4d 2h
- Merged PRs (30d)
- 12
Description
## Summary
A program that reads an object over the ROS3 (S3) VFD and then exits normally hangs forever during process shutdown on Windows. All application work completes successfully (files open, read, and close fine); the process simply never exits.
A native stack captured from a minidump of the hung process shows the main thread blocked in the HDF5 S3 cleanup that runs at process exit, trying to join AWS worker threads while the Windows loader lock is held.
This does not reproduce on Linux or macOS. This was reported previously in https://github.com/h5py/h5py/issues/2850 and separately caused downstream CI issues in https://github.com/NeurodataWithoutBorders/pynwb/issues/2228 which I maintain.
_Disclaimer:_ This issue was investigated using AI (Claude Opus 4.8) to help narrow down the root cause and suggest fixes. AI was used to analyze the native stack, understand the AWS C SDK behavior, and draft this issue report. I reviewed and edited the final content to the best of my ability, but I do not have a deep understanding of the HDF5 or AWS C SDK internals, so please verify the analysis and proposed fixes.
## Environment
- HDF5 2.1.0 (conda-forge build, ROS3 VFD backed by the AWS C SDK)
- aws-c-s3 0.12.8, aws-c-common 0.14.2, aws-c-io 0.27.3, aws-c-auth 0.10.4 (conda-forge)
- h5py 3.16.0, Python 3.14 (but the deadlock is in native teardown and is not Python-specific)
- Windows Server 2025 (GitHub Actions `windows-latest`)
## Reproduction
Open any S3 object with the `ros3` driver on Windows, then let the process exit:
```python
import h5py
url = "https://dandiarchive.s3.amazonaws.com/ros3test.nwb"
with h5py.File(url, mode="r", driver="ros3", aws_region=b"us-east-2") as f:
_ = f # read/close all succeed
# On Windows the process now hangs here, during interpreter/process exit.
```
The read completes and the file closes cleanly; the hang is purely at process teardown.
## Native stack of the hung main thread
Captured with `procdump -ma ` and `cdb -z dump.dmp -c "~*k"`. The `hdf5!...` frame symbols are approximate (nearest exported symbol + offset) because the conda-forge `hdf5.dll` ships without PDBs; the `aws_c_*` and `ntdll!Ldr*` frames are exported and unambiguous.
```
ntdll!NtWaitForAlertByThreadId
ntdll!RtlSleepConditionVariableSRW
KERNELBASE!SleepConditionVariableSRW
aws_c_common!aws_condition_variable_wait
aws_c_common!aws_condition_variable_wait_pred
aws_c_common!aws_thread_join_all_managed <- blocks joining AWS worker threads
aws_c_s3!aws_s3_library_clean_up <- AWS-C-S3 library teardown
hdf5! (ROS3 / H5FD s3comms cleanup) <- approximate symbol
ucrtbase!execute_onexit_table <- hdf5.dll CRT atexit table
ntdll!LdrpCallInitRoutine
ntdll!LdrShutdownProcess <- DLL_PROCESS_DETACH, loader lock held
ntdll!RtlExitUserProcess
kernel32!ExitProcessImplementation
ucrtbase!common_exit
python314!Py_Exit
...
python314!Py_RunMain
```
## Claude's hypothesized root cause
HDF5 registers the AWS-C-S3 library teardown (`aws_s3_library_clean_up`) to run at process exit via the C runtime's `onexit`/`atexit` table in `hdf5.dll`. On Windows, that table runs inside `DLL_PROCESS_DETACH`, which the loader executes **while holding the loader lock**. `aws_s3_library_clean_up()` calls `aws_thread_join_all_managed()`, which blocks waiting to join AWS-managed worker threads. Those threads cannot make progress to completion while the loader lock is held, so the join never returns and the process deadlocks.
Joining threads during `DllMain`/`DLL_PROCESS_DETACH` is not supported on Windows; this is a classic loader-lock deadlock.
### Why it is Windows-only
Linux and macOS join threads at exit without a loader-lock-during-detach constraint, so the same cleanup completes normally.
### Why it appeared with HDF5 2.x
HDF5 2.x's ROS3 VFD is backed by the AWS C SDK (`aws-c-s3` / `aws-c-common`), which spawns managed worker threads and registers the join-on-exit cleanup. The earlier libcurl-based ROS3 VFD spawned no such threads, so exit was clean.
### Corroborating observation
In h5py's test suite (`h5py/tests/test_ros3.py`), the tests that call `h5py.File(..., driver="ros3")` to open or attempt to open an S3 location **hang on exit**: `test_ros3` (opens a real object) and `test_ros3_s3_fails` (attempts an open against a nonexistent bucket, which still initializes the S3 VFD). `test_ros3_temp_token`, which only builds and queries a file-access property list (`set_fapl_ros3` / `get_fapl_ros3_token`) without opening a file, does not hang. This split is reported in https://github.com/h5py/h5py/issues/2850 and is consistent with the deadlock being in `aws_s3_library_clean_up` joining threads that exist only once the AWS-C-S3 library has been initialized by an S3 open.
### What does not work as a workaround
`os._exit()` / `ExitProcess` do **not** avoid the hang, because on Windows they still run `DLL_PROCESS_DETACH` and hit the same deadlock. Only terminating the process from an external parent process (which is not itself running the loader teardown) avoids it.
## Claude's suggested fixes
- Do not join threads during process-exit DLL detach on Windows. For example, skip the `aws_thread_join_all_managed()` path when running under `DLL_PROCESS_DETACH`, or
- Do not register `aws_s3_library_clean_up` to run via `atexit`/DLL detach at all; instead tie AWS-C-S3 init/cleanup to explicit `H5FDros3`/S3 VFD driver registration/termination or to `H5close()`, so cleanup happens before process exit rather than during loader teardown.
Contributor guide
Assessment
This issue has not been assessed yet.