HarperFast / HarperFast/harper

Windows: set_configuration returns 500 because the config write's rename retry blocks the thread that would release the handle

Open
#2,313 1 comment 0 reactions 1 assignee Claimed by @kriszyp View on GitHub
bug
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 2h
Merged PRs (30d)
205

Description

## Summary

On Windows, `set_configuration` intermittently returns HTTP 500. The atomic config write fails
with `EPERM: operation not permitted, rename`, and the operation's thread is stalled for the
full ~3.6 s retry budget before the error surfaces.

Seen on `main`: https://github.com/HarperFast/harper/actions/runs/32809913865, job
`Integration Tests 6/6 (Windows, Node.js v24)`, failing
`integrationTests/apiTests/configuration.test.mjs:319`
(`expected 200 "OK", got 500 "Internal Server Error"`).

```
2026-08-25T04:46:04.396Z [main/0] [error]: Error: EPERM: operation not permitted, rename
...\harper-config.yaml.6828.0.cc696ba1.tmp -> ...\harper-config.yaml
at atomicWriteFile (dist/config/configUtils.js:168)
at updateConfigValue -> setConfiguration
```

## Root cause

The retry loop is self-defeating: it waits out the holder by blocking the very thread that must
run to release it.

1. Windows `rename()` over an existing destination fails while any descriptor is open on that
destination.
2. Harper's own root-config watchers are that holder. Every thread runs a `RootConfigWatcher`
(`utility/logging/harper_logger.ts`) and one `OptionsWatcher` per root-config component scope
(`components/Scope.ts`); each reacts to a change with `fsPromises.readFile` on
`harper-config.yaml` (`config/RootConfigWatcher.ts`, `components/OptionsWatcher.ts`). That
read's `close()` continuation runs on the owning thread's event loop, so the descriptor is
held across event-loop turns. One config write fans out dozens of these reads.
3. `atomicWriteFile` (`config/configUtils.ts`) retries `renameSync` with `Atomics.wait` on the
calling thread. When the holder is an in-flight read on that same thread, its `close()` can
never be scheduled while we wait — so the holder's lifetime is exactly the retry budget and
every attempt is guaranteed to fail.

The trigger is back-to-back writes. In the failing run the first `set_configuration` at
`04:46:00.610` added a component key (the largest possible watcher fan-out — it creates a new
scope); the second at `04:46:00.638`, 28 ms later, started while those reads were in flight and
errored at `04:46:04.388`. The backoff sums to exactly 3630 ms; the observed gap is 3750 ms, and
the main thread logs nothing at all in between.

This also explains the fix history: #1714 and #2036 each widened this budget. Neither could work
— widening it only lengthens the failure.

### Measured on the Windows CI runner

Node v24.19.0, `windows-latest`, reproduced identically on two independently dispatched runs
([1](https://github.com/HarperFast/harper/actions/runs/32845524581),
[2](https://github.com/HarperFast/harper/actions/runs/32845782116)):

| case | result |
|---|---|
| control: rename over destination, no handles open | ok |
| **destination held by a single Node `fs.openSync(dest, 'r')` descriptor** | **EPERM** |
| same, immediately after closing that descriptor | ok |
| **destination held by an in-flight `fsPromises.readFile` while the production retry loop blocks the thread** | **EPERM on all 13 attempts** |
| same, immediately after awaiting that read | **ok** |
| source (the `.tmp`) held open by a descriptor | ok |
| destination watched by `fs.watch` (file), `fs.watch` (dir), or chokidar | ok |
| chokidar watching + in-flight read, loop blocking | **EPERM on all 13 attempts** |

Three consequences, all measured rather than assumed: a Node read descriptor on the destination
**does** block Windows rename despite libuv opening files with `FILE_SHARE_DELETE`; the *watch*
handle is innocent, which is why the failure is intermittent rather than permanent; and holding
the source open does not block, so antivirus scanning the freshly written temp file is not the
mechanism.

## Impact

- `set_configuration` returns 500 and the config change is not applied. The write stays atomic —
the previous config is intact and the temp file is cleaned up — so this is a failed operation,
not corruption.
- The operation's thread is blocked for ~3.6 s per occurrence, so every other request on that
thread stalls with it.
- Windows only. POSIX `rename` over an open file is legal, so Linux and macOS are unaffected.
- Present in shipped versions: the retry loop has been in place since #1714.

## Fix direction

Bound the descriptor to a single syscall — read the root config synchronously in both watchers,
so no config read outlives the event-loop turn it started in. Application configs are written in
place (`fs.outputFile`), never by rename-over, so they keep the non-blocking read.

Related: #1143 (atomic config write hardening), #1747 (a config write lost at the watcher layer),
#525 (Windows integration testing).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.