danielmiessler / danielmiessler/LifeOS

Pulse serves stale code indefinitely after a source change, and cannot exit cleanly to reload

Open
#2,101 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
19k
Forks
2.5k
Avg merge
8d 17h
Merged PRs (30d)
1

Description

### Version

LifeOS 7.40.4 (verified against `5e2f2e8`)

### What is broken

Two related things in the Pulse daemon. First, Pulse never notices that its source has moved ahead of the running process. It is a long-lived launchd job with `KeepAlive`, so editing anything under `LIFEOS/PULSE/` changes the file and nothing else: the process keeps serving whatever it loaded at start, and no log line, health field or dashboard tile says the two have diverged. A fix committed to a Pulse file therefore has no effect until somebody remembers to restart the daemon, and if nobody does, the original symptom keeps being logged after the bug is marked fixed. Second, and only visible once something tries to act on the first, returning from `main()` does not end the process. The cleanup block closes the HTTP server and fires the module stops without awaiting them, so a polling loop can hold the event loop open, and there is no explicit exit afterwards. Nothing notices today because every stop comes from launchd, which follows its `SIGTERM` with a `SIGKILL`. Anything that tries to exit deliberately instead gets a daemon that survives with its server already closed: listening on nothing, and never restarted, because from launchd's point of view it is still running.

### Where (file:line)

`LIFEOS/PULSE/pulse.ts:1208-1214` (cleanup closes the server, fires `stopIMessage` and `stopAssistant` unawaited, then returns with no exit; the only `process.exit` on the success path is absent, line 1218 being the crash handler). The staleness gap is the absence of any check in the cron loop that begins at `LIFEOS/PULSE/pulse.ts:597`.

### Repro on a clean tree

```shell
# 1. Staleness. Start Pulse however the install normally does, then note the pid.
launchctl list com.lifeos.pulse # or: cat LIFEOS/PULSE/state/pulse.pid

# 2. Add an unmistakable line to a file the daemon loaded at start.
echo 'log("info", "STALENESS PROBE")' >> LIFEOS/PULSE/modules/wiki.ts

# 3. Wait past a full cron tick, then look for it.
grep -c 'STALENESS PROBE' LIFEOS/PULSE/logs/pulse-stdout.log
# 0 — indefinitely, and the pid is unchanged

# 4. The only thing that picks it up is a manual restart.
launchctl kickstart -k gui/$(id -u)/com.lifeos.pulse
```

For the second part, the observable shape is the cleanup block itself: `server.stop()` runs, `stopIMessage`/`stopAssistant` are fired without `await`, and `main()` returns without `process.exit`. Under launchd this is invisible because the kill arrives regardless.

### Negative control

On unpatched 7.40.4, step 3 above returns `0` for as long as the daemon runs, with the process start time never changing, while the source on disk contains a line the running code does not have. There is no warning anywhere in the log, so the state is indistinguishable from a correctly updated daemon.

For the exit path, my evidence is from a fork rather than this tree, and I would rather say so than dress it up: adding a staleness check to the cron loop that set the shutdown flag and broke out of it produced a process that logged `LifeOS Pulse stopped`, closed port 31337, and then stayed alive indefinitely with launchd still counting it as running. The daemon served nothing until it was kickstarted by hand. The cleanup block in this tree is the same shape, minus the module whose polling held the loop open, so I can confirm the missing exit but not that this tree has a holder today.

### Suggested fix

For the staleness gap, `WatchPaths` is the obvious answer and I would argue against it: `man 5 launchd.plist` calls it highly discouraged, filesystem monitoring is race-prone so modifications can be missed, and a caught modification gives no guarantee the file is consistent when the job relaunches, which against `KeepAlive` is a throttled crash loop. It would also need an entry per directory and would miss imports from outside the Pulse tree.

Asking the question inside Pulse works better, on the cron tick that already runs: walk the module graph reachable from the entrypoint by relative import, compare the newest mtime against `state.startedAt`, and when the source is ahead, log which file and exit so launchd reloads on the new code. That follows the real dependency graph rather than a hand-maintained directory list, never fires on a half-written file because the daemon chooses its own moment, and self-corrects on the next tick if a change is missed. A small tolerance window stops a write moments before launch reading as newer than the process it was already loaded by, and deferring while a message turn is in flight stops it dropping a reply someone is waiting on.

That fix depends on the exit being explicit, which is the other half of this report.

Running in a fork with 12 tests over the graph walk and the stale decision; measured detection latency is bounded by the existing tick sleep, and relaunch took 1.1 seconds. Happy to open a PR if it would be useful.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with LIFEOS/PULSE/pulse.ts:597 and trace the cron loop, then inspect the cleanup block at lines 1208-1214 and the related module stop calls. Use the reported stale-source and deliberate-exit repros as the baseline, with the 12 graph-walk and stale-decision tests from the fork as a testing reference. Done means changed reachable source is detected and logged, the daemon exits cleanly, and launchd can reload it without manual intervention.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.