ActivityWatch / ActivityWatch/aw-watcher-window

[macOS] aw-watcher-window-macos leaks AXObserver/Mach ports, causing high wired memory

Aperta
#139 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
129
Fork
81
Merge medio
2g 10h
PR unite (30g)
4

Descrizione

## Summary

On macOS, `aw-watcher-window-macos` appears to leak Accessibility (`AXObserver`) receive ports during long-running use.

After ActivityWatch 0.13.2 had been running for about 16 days, the helper owned **7,339 Mach receive ports** containing **3,791,856 queued messages**. This represented **99.897% of all queued Mach messages on the system**.

Quitting ActivityWatch caused approximately **3.807 million related IPC, VM map copy, VM object, and compressor objects to disappear immediately**, confirming that the accumulated kernel objects belonged to the ActivityWatch window watcher.

The accumulation resulted in approximately **6.1 GiB of wired memory** on a Mac with 18 GiB unified memory.

## Environment

- ActivityWatch: 0.13.2
- macOS: 26.5.2 (25F84)
- Hardware: Apple M3 Pro, 18 GiB unified memory
- Window watcher strategy: Swift
- Helper: `aw-watcher-window-macos`
- Helper architecture: x86_64 running through Rosetta
- Uptime when diagnosed: approximately 16 days

I have not yet tested the current prerelease build.

## Observed behavior

Wired memory remained around 6.1–6.3 GiB even when no VM, container, local LLM, or other obvious high-wired workload was running.

`zprint` showed several related kernel zones with approximately 3.8–4.8 million live objects:

```text
VM.map.entries 4,840,900
VM.map.copies 3,832,294
ipc.kmsgs 3,815,571
vm.objects 4,271,204
data.kalloc.192 3,767,849
compressor_pager 4,068,152
compressor_slots.64 3,877,006
```

Between two samples taken approximately five hours apart, the following zones grew by nearly identical amounts:

```text
ipc.kmsgs +50,788
VM.map.copies +49,823
data.kalloc.192 +50,661
```

A 30-second idle sample was flat, suggesting that the accumulation is event-triggered, possibly by switching applications or focused windows, rather than a fixed-rate background loop.

## Mach port evidence

I captured a system-wide Mach port snapshot using Apple's `lsmp`:

```bash
sudo /usr/bin/lsmp -a -j /tmp/lsmp.json
```

The system contained 3,795,755 queued messages in total. The top process was:

```text
Process: aw-watcher-window-macos
PID: 4669
Total port names: 7,428
Receive ports: 7,339
Non-empty receive ports: 6,967
Queues at their limit: 2,853
Queues with 1024 messages: 2,852
Queued messages: 3,791,856
System queue share: 99.8973%
```

The median non-empty queue contained 522 messages. P90 and P99 were both 1024, so thousands of queues had reached their limit.

Other applications held send rights to the affected port objects. For example:

```text
Google Chrome: 2,125 matching ports
iTerm2: 1,477 matching ports
BetterCmdTab: 606 matching ports
Obsidian: 384 matching ports
Typora: 221 matching ports
```

These applications appear to be Accessibility event sources. The leaked receive rights and queued messages were owned by `aw-watcher-window-macos`, so the sender applications do not appear to be the leak owners.

`lsmp` only proves that these applications held send rights; it does not provide per-message sender attribution.

## Controlled shutdown test

I completely quit ActivityWatch and immediately repeated the kernel-zone measurements.

```text
Zone Before quit After quit Reduction
ipc.kmsgs 3,815,571 6,749 99.82%
VM.map.copies 3,832,294 23,339 99.39%
data.kalloc.192 3,767,849 3,928 99.90%
vm.objects 4,271,204 463,979 89.15%
VM.map.entries 4,840,900 1,104,452 77.18%
compressor_pager 4,068,152 260,719 93.59%
compressor_slots.64 3,877,006 69,846 98.20%
```

Approximately 3.807 million related objects disappeared together.

Wired pages initially measured 398,764 pages, or approximately 6.08 GiB with 16 KiB pages. After quitting ActivityWatch, the value began to decline and briefly reached approximately 5.53 GiB.

Not all previously expanded kernel-zone pages were returned immediately, which is expected after such a large long-running accumulation. A reboot should fully reclaim the remaining expanded zone/slab memory.

## Possible source-level cause

The current macOS Swift implementation is here:

https://github.com/ActivityWatch/aw-watcher-window/blob/master/aw_watcher_window/macos.swift

Two cleanup paths look suspicious:

1. A window is registered for `kAXTitleChangedNotification`, but when replacing `oldWindow`, the code appears to remove `kAXFocusedWindowChangedNotification` from the old window instead. The removed notification does not match the notification that was added.

2. When the focused application changes, the old observer's RunLoop source is removed, but the old application/window notification registrations do not appear to be explicitly removed before `AXObserverCreate` replaces the observer.

This could leave old AX observers or notification registrations alive, with the observed applications retaining send rights to abandoned Mach receive ports. Incoming AX notifications would then accumulate until each queue reaches its 1024-message limit.

This source-level explanation is currently a hypothesis. The installed 0.13.2 binary was not disassembled, and the current `master` source may differ from the exact source bundled in 0.13.2. The Mach port ownership and shutdown test, however, directly identify `aw-watcher-window-macos` as the owner of the accumulation.

## Expected behavior

- Notifications registered on an old window should be removed using the same notification type that was added.
- Previous application and window notifications should be unregistered before replacing an `AXObserver`.
- The number of Mach receive ports and queued messages should remain bounded during long-running use.
- Running ActivityWatch for several days should not cause wired memory to grow continually.

## Actual behavior

Thousands of receive ports accumulate over time, many queues reach 1024 messages, and the queued Mach/OOL messages retain millions of VM and compressor objects.

## Workaround

Completely quitting and restarting ActivityWatch releases the active objects. However, kernel zone/slab memory may not be fully reclaimed until macOS is restarted.

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.