pingdotgg / pingdotgg/t3code

[Bug]: Backend event loop blocks for minutes inside synchronous fff mixedSearch waiting on watcher lock (home-directory project) — shows as "<label> is reconnecting"

Open
#11,306 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
23k
Forks
5.9k
Avg merge
11h 14m
Merged PRs (30d)
357

Description

Before submitting
  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.
Area

apps/server

Steps to reproduce
  1. Add the user's home directory (/Users/<name>) as a T3 project on macOS. Home contains ~25 GB under Library/Caches plus constantly written directories (.t3, .claude, .ccs).
  2. Run several concurrent agent threads so the backend is busy and files under home change continuously.
  3. Open the Files panel for that project, or let the Files panel refresh.
  4. Watch the composer banner.
Expected behavior

A synchronous file-search call should not be able to block the backend event loop for minutes. A busy file watcher should not make the Files panel call hang.

Actual behavior

The desktop backend stops responding entirely: no WebSocket frames, no trace spans, no provider log writes, health endpoint unreachable. The desktop shows "<label> is reconnecting / Trying again" and "Not connected: message not sent". The freeze lasts 1 to 6 minutes, then either recovers on its own or the user relaunches the app.

Six freezes in ~20 hours on 0.0.41-nightly.20260911.1520 (and .20260910.1507), each visible as a gap in server.trace.ndjson where every span source stops at once:

Freeze start (UTC) Duration Ended by
00:51:03 90s user relaunch
01:08:41 55s updater relaunch
20:20:09 385s recovered
20:35:43 157s recovered
Root cause (from sample of the backend during the 20:35 freeze)

All 4099 main-thread samples show the Node event loop blocked inside a synchronous FFI call into libfff_c, waiting on a parking_lot read lock. The lock is held by fff's own watcher thread, which is reading files to sniff for binary content.

Thread main (100% of samples):
  uv_run -> uv__work_done -> node::fs::AfterStat -> JS callback
    -> ffi-rs -> fff_search_mixed
      -> parking_lot::raw_rwlock::RawRwLock::lock_shared_slow
        -> __psynch_cvwait

Thread fff-watcher-own (holds the lock):
  fff_search::file_picker::FilePicker::add_new_file
    -> fff_search::types::FileItem::detect_binary_per_byte -> read()

Thread "notify-rs debouncer loop":
  fff_search::background_watcher::handle_debounced_events
    -> RawRwLock::lock_exclusive_slow   (also waiting)

Thread fff-scan:
  fff_search::scan::ScanJob::run
    -> bigram_filter::sniff_binary_for_non_indexable -> open()/read()

Two things combine:

  1. apps/server/src/workspace/WorkspaceSearchIndex.ts calls finder.mixedSearch(...) (line ~236) and finder.isScanning() synchronously on the event loop. Any lock contention inside the native library stalls the whole server.
  2. The finder is created with enableFsRootScanning: true, enableHomeDirScanning: true (line ~172), so a home-directory project indexes and watches the entire home tree. Every burst of writes under home (agent transcripts, caches, T3's own logs) makes the watcher take the write lock and read each new file while a listEntries / search call waits.

The 25,000-entry listEntries call for the home project takes ~13 s even when it does not deadlock (WorkspaceSearchIndex.waitForIndexReady 12993 ms in the trace).

Workaround

A ~/.ignore excluding Library/, .cache/, .t3/, .claude/, .ccs/, node_modules/ etc. shrinks the watched set (the ignore crate honors .ignore outside git). Removing the home project stops it entirely.

Suggested fixes
  • Run mixedSearch / isScanning off the event loop (worker thread or async FFI), or bound them with a timeout that fails the single request instead of the process.
  • Do not default enableHomeDirScanning / enableFsRootScanning to true. Or apply a large default exclusion set when the workspace root is $HOME or /.
  • Related: #10918 (no way to exclude directories), #10917 (hardcoded 15 s scan timeout), #7231 (a slow probe tearing down the session, which is the client-side symptom of this).
Environment
  • T3 Code (Nightly) 0.0.41-nightly.20260911.1520, also on .20260910.1507
  • macOS 26.4 (25E246), Apple Silicon (M5)
  • @ff-labs/fff-node 0.9.4
  • Local desktop backend, direct connection, no remote environment
  • state.sqlite 1.9 GB (not the cause, noted for context)

Full sample output available on request (269 KB). Diagnosed with Claude Fable 5.1 in T3 Code.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in apps/server/src/workspace/WorkspaceSearchIndex.ts, especially the synchronous finder.mixedSearch and finder.isScanning calls around line 236 and finder setup around line 172. Reproduce or inspect the waitForIndexReady trace under a home-directory project with active file changes; done means the backend remains responsive and a slow search cannot block it for minutes.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.