Tool calls block other users

Async evaluation: why the app hangs under concurrent use

Investigation of why the tiles agent can freeze for all users when someone runs a
slow query, and what an async fix would look like. Status: investigation only —
no implementation yet. Recommended direction recorded at the end.

Symptom

From testing: occasionally, near the start of a conversation, the app "sits there
forever" with the progress indicator, then after ~a minute quickly performs tool
calls and streams results. Time-to-first-token is very long, but streaming is fine
once it starts. Concern: a query that genuinely takes a minute, run synchronously,
would block every other session.

That concern is correct. Here is the mechanism.

The Shiny plumbing is already correct

The blocking is not a misuse of ExtendedTask or shinychat. The chain is:

• commons::commons_mod_server() (commons/R/chat.R:50) delegates to
  shinychat::chat_mod_server().
• shinychat::chat_mod_server() already wraps each turn in
  shiny::ExtendedTask$new(function(client, ui_id, user_input) { stream <- client$stream_async(user_input, stream = "content"); ... }).
• client$stream_async() (ellmer) streams the LLM's HTTP response via httr2's async
  curl handle + later. This genuinely yields to the event loop between chunks,
  so token streaming does not block other sessions.

So the LLM network legs are already non-blocking.

The LLM connections must NOT be mirai-wrapped

A natural first instinct is "wrap the LLM calls in mirai too." That is wrong, and
worth recording. ellmer's chat_perform_async_stream performs each LLM request as a
non-blocking socket and cooperates with the event loop directly:

resp <- req_perform_connection(req, blocking = FALSE)
repeat {
  event <- chat_resp_stream(provider, resp)
  if (is.null(event) && !resp_stream_is_complete(resp)) {
    fds <- resp$body$get_fdset()
    await(promises::promise(function(resolve, reject) {
      later::later_fd(resolve, fds$reads, fds$writes, fds$exceptions, fds$timeout)
    }))
    next
  }
  ...
}

When no data is on the socket yet, it registers the socket file descriptors with the
event loop (later::later_fd) and awaits — handing control back to Shiny until
bytes arrive. This is cooperative async at the socket level.

The distinction that matters:

• mirai is for work that blocks the R thread and cannot yield — synchronous C
  calls (ODBC/Redshift dbGetQuery) and CPU-bound R. You move it off-process because
  it cannot cooperate with the event loop.
• httr2 async (ellmer's LLM path) is for I/O-bound network calls; it already
  cooperates with the event loop via later_fd. No separate process needed.

Wrapping an LLM call in mirai() would gain nothing (already non-blocking) and would
lose token streaming — the daemon would run the request to completion and return
only the final text. So the LLM path stays as-is; all async work is on the DB-tool
side.

The actual blocker: synchronous tool execution

During a turn, ellmer runs tools via invoke_tool_async (ellmer namespace):

result <- await(do.call(request@tool, args))

await() only yields to the event loop if the tool returns a promise. Every
commons tool is a plain synchronous function (commons/R/tools.R):

• run_sql -> run_sql_tool() -> source_query() ->
  DBI::dbGetQuery(source$con, sql) (commons/R/data-source.R:240). A blocking
  Redshift query on the main R process.
• call_measure -> call_measure_tool() -> do.call(td, ...). Runs a measure
  synchronously; measures themselves query the warehouse.
• describe_table -> describe_table_tool() -> blocking dbGetQuery.
• search_measures / search_context -> in-memory BM25; fast, not a concern.

R/Shiny is single-threaded. While a synchronous tool runs, the entire event loop is
frozen for every connected session until the call returns. A 60-second warehouse
query is a 60-second freeze for everyone.

This also explains the long time-to-first-token: the agent typically does one or more
synchronous tool rounds (context search + SQL/measure) before it emits any streamed
assistant text. The wait is those blocking tool calls, not the LLM.

Secondary blocker: session startup

The app's Shiny server() builds the agent — opening the warehouse connection,
connecting the pins board, and constructing the agent (dictionaries, pins, BM25
index) — synchronously on every new session. That also blocks existing sessions while
a new user connects. Lower priority than tool execution, but the same single-threaded
hazard.

The core challenge for any fix

source$con is a live ODBC/Redshift DBI connection created in the main process. It
cannot be serialized to a mirai daemon. So async-ifying is not just "wrap the call
in mirai()". Either:

1. Each daemon establishes and holds its own warehouse connection, and we ship SQL
   strings (serializable) to it; or
2. We run the whole turn in a per-session daemon that owns the connection — which
   loses token-by-token streaming.

Because these tools live in commons, the real fix is a commons change, not a
tiles-agent-only patch.

Options considered

Option A (recommended) — offload tools to a daemon pool, keep streaming

Keep LLM streaming async on the main process (already works). Make the blocking tools
(run_sql, call_measure, describe_table) async: the tool function returns a
mirai promise, so ellmer's await(do.call(...)) yields and the event loop stays free.

Sketch:

• A daemon pool where each daemon opens its own warehouse connection once:

  daemons(n, .compute = "warehouse")
  everywhere(con <- warehouse::lakehouse(), .compute = "warehouse")
  

• run_sql tool dispatches a SQL string and returns a promise:

  mirai(DBI::dbGetQuery(con, sql), .args = list(sql = sql), .compute = "warehouse")
  

  mirai auto-converts to a promise, which ellmer awaits without blocking.
• Measures are harder: a measure is an R function that needs dscoetools loaded, the
  measure body available on the daemon, and a daemon-local connection injected in
  place of the main-process warehouse argument. commons already injects warehouse
  by name via data_sources; the daemon variant would inject the daemon's own con.

Pros: preserves token streaming and the current UX; unblocks the event loop during DB
work; benefits every commons app.
Cons: most engineering. Requires a "reconnect-in-daemon" source abstraction, measure
injection onto daemons, daemon-pool lifecycle (onStop(function() daemons(0))), and
verifying warehouse auth works inside a daemon on Connect/Workbench (see open
question below).

Option B — run the whole turn in a per-session daemon

Each session gets a mirai daemon owning the ellmer Chat + connection; ExtendedTask
runs the full synchronous chat() there and returns the final text.

Pros: simplest to reason about; fully non-blocking; no per-tool async refactor.
Cons: loses token-by-token streaming (spinner + final render only); diverges from
shinychat's streaming module; one process per concurrent user.

Option C — status quo

A single slow query freezes the whole app. Acceptable only at very low concurrency.

Recommended direction

Option A (offload tools, keep streaming), implemented in commons. It is the
correct long-term home and preserves the streaming UX. It is also the larger change,
so it warranted a spike before committing — see below.

Spike results (2026-07-10) — PASSED on Workbench AND Connect

The make-or-break spike passed in both environments, so Option A is cleared.

Connect (a throwaway R Markdown diagnostic deployed to Connect): the sandbox
spawns mirai daemons (mirai(1 + 1) returned 2), the warehouse env vars propagate
to the daemon, warehouse loads there, and warehouse::lakehouse() authenticates +
queries inside the daemon (select 1 -> 1). So Connect's credential mechanism
reaches child processes and subprocess spawning is allowed — the two things Workbench
couldn't tell us.

Workbench (an interactive spike): passed, with the fuller secondary checks below.

• Env vars propagate. The warehouse env vars reach daemons.
• Concurrency works. Two 2s queries on two daemons finished in ~2s (parallel), not
  ~4s (serial).
• Connection stability. A daemon-held connection survived 60s idle.
• Measures run on daemons. new_hidi_leads() returned 193 rows from a daemon with
  a daemon-local connection.
• Cancellation returns fast. stop_mirai() returned in ~1.5s and didn't block.
• Startup overhead. ~2s warmup on first query after everywhere() (package load +
  connect); subsequent queries fast.

Critical implementation detail — connection scoping in everywhere(). Assigning
with con <- warehouse::lakehouse() inside everywhere() puts con in a scope that
later mirai() calls cannot see. The connection must land in the daemon's global
environment:

everywhere({
  library(warehouse)
  .GlobalEnv$con <- warehouse::lakehouse()   # NOT `con <- ...`
})

(con <<- warehouse::lakehouse() would also reach the global env.) This is because a
mirai() expression evaluates in a fresh local env whose parent is the daemon's global
env, so shared objects must live in the global env.

Open questions to resolve before implementing

Feasibility is settled (both environments pass). What remains is implementation
design, not go/no-go:

• Server-side cancellation. stop_mirai() returns fast, but whether the underlying
  Redshift query actually terminates server-side (vs. keeps running) was not verified —
  check the warehouse's running-queries view.
• Pool sizing / lifecycle. Shared pool across sessions vs per-session daemons; how
  many daemons; cleanup on session end and app stop (onStop(function() daemons(0))).
  Startup is ~2s/daemon, so a 4-daemon pool adds noticeable boot time.
• Measure execution wiring in commons. The spike proved a measure can run on a
  daemon; the commons change still needs to ship the measure fn + dscoetools + a
  daemon-local connection in place of the current main-process warehouse injection.
• Connect deploy impact. New soft-deps (mirai, nanonext) would need to be
  pinned so Connect's dependency scanner bundles them.

Key references

• commons/R/chat.R:50 — commons_mod_server delegating to shinychat.
• commons/R/tools.R — all tools; synchronous run_sql_tool / call_measure_tool /
  describe_table_tool.
• commons/R/data-source.R:240 — source_query() = DBI::dbGetQuery(source$con, sql).
• ellmer invoke_tool_async — await(do.call(request@tool, args)).
• shinychat chat_mod_server — ExtendedTask + stream_async.