electric-sql / electric-sql/electric
RFC: `signal` verb
- Dominant language
- TypeScript
- Stars
- 10.4k
- Forks
- 375
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 18
Description
## Summary
Agents need the same lifecycle controls that Unix gave processes 50 years ago. A user hits "stop" and the agent should drop what it's doing immediately. A deploy ships and every running entity should pick up the new code. An entity misbehaves and an operator needs to kill it, pause it, or let it clean up gracefully. Custom signals let developers build domain-specific interrupts — cancel a workflow, reprioritize a task, inject new instructions mid-run.
The signal verb gives agents/entities all of this, directly modeled on Unix signals: SIGINT, SIGTERM, SIGKILL, SIGSTOP, SIGCONT, SIGHUP, SIGUSR.
## Signal Delivery
The server is just a write path — it inserts a `signal` event into the entity's StreamDB. The runtime reacts.
```
POST /{entity_type}/{instance_id}/signal
│
▼
Server inserts signal ────────── Written to entity's StreamDB
│
▼
TanStack DB collection updates ────────── Runtime's materialized view
│
▼
Registered effect fires ──────────────── Immediate reaction to signal insert
│
▼
Validate signal ──────────────────────── Query collection for current state
│ (e.g., can't pause what's already paused)
│
├── Invalid? ──── Reject signal
│
▼
Handle signal ────────────────────────── Write state transition if applicable
│ (e.g., paused, stopping, killed)
│ Or act directly (SIGINT, SIGHUP, SIGUSR)
│
▼
Enforce behavioral change ────────────── Stop message delivery, start timer, abort runs, etc.
```
Some signals are immediate — SIGINT, SIGKILL, and SIGUSR act as soon as they land in the TanStack DB collection, even mid-run. The rest (SIGTERM, SIGSTOP, SIGCONT, SIGHUP) are checked between LLM API calls, tool calls, and messages. A tool call that takes five minutes will finish before a non-immediate signal is checked.
As in Unix, entity code can register handlers for any signal except SIGKILL and SIGSTOP, which the runtime enforces unconditionally. The `onSignal` hook receives the signal type and payload.
### Crash recovery
On runtime restart, TanStack DB collections are rebuilt from the entity's StreamDB. Effects re-register. The StreamDB is the durable source of truth.
### Concurrent signal handling
If two signals arrive simultaneously for the same entity:
1. Server inserts both `signal` events into the StreamDB
2. The runtime's effects fire for each
3. The first effect validates and writes the state transition
4. The second effect reads the now-updated state from the collection
5. If the signal is still valid, it proceeds; if not (e.g., entity is now stopped), it's rejected
The StreamDB serializes writes. The collection always reflects the latest state.
## Signal Event Format
Signal events follow the standard event format on the entity's StreamDB:
```json
{
"type": "signal",
"key": "sig_001",
"value": {
"signal": "SIGTERM",
"sender": "/darix/runtime",
"reason": "user requested shutdown"
},
"headers": {
"operation": "insert",
"timestamp": "2026-03-07T10:00:00Z"
}
}
```
When a signal changes lifecycle state, the runtime writes the new state (`paused`, `running`, `stopped`, `killed`) to the StreamDB. SIGUSR does not change state — only the signal event is written.
## API
`POST /{entity_type}/{instance_id}/signal`
```json
{
"signal": "SIGTERM",
"reason": "user requested shutdown"
}
```
Response:
```json
{
"url": "/my_agent/agent_1",
"signal": "SIGTERM",
"previous_state": "running",
"new_state": "stopping",
"created_at": 1741334400000,
"txid": "tx_001"
}
```
Error (server rejects — entity in terminal state):
```json
{
"error": {
"code": "INVALID_SIGNAL",
"message": "Cannot signal a stopped entity"
}
}
```
### Migration: DELETE → SIGKILL
`POST /{entity_type}/{instance_id}/signal` with `"signal": "SIGKILL"` replaces `DELETE /{entity_type}/{instance_id}`. Migrate all existing DELETE calls.
## Entity Lifecycle State Machine
```
spawning ──► running ◄──► idle (runtime shuts down after idle timeout)
│ │
│ └──► paused ──► running (via SIGCONT)
│ │
│ ▼
│ stopping ──► stopped (via SIGTERM from paused)
│
├──► stopping ──► stopped (via SIGTERM grace period expiry or early completion)
│
└──► killed (via SIGKILL, from any non-terminal state)
stopped / killed ──► streams closed (EOF) but retained for replay/audit
```
This spec introduces three states: `paused`, `stopping`, and `killed`. `idle` exists in the current system. `stopping` is a transitional state for the SIGTERM grace period.
`stopping → stopped` is not signal-driven. The entity moves to `stopped` when the grace period expires or the handler finishes cleanup early.
Signal handling by state:
Server rejects signals for entities in terminal states (`stopped`, `killed`) — the signal is never written to the StreamDB. For all other states, the server writes the signal to the StreamDB and the runtime handles it.
| Current State | SIGINT | SIGHUP | SIGTERM | SIGKILL | SIGSTOP | SIGCONT | SIGUSR |
|---|---|---|---|---|---|---|---|
| `spawning` | ignored | ignored | ignored | → `killed` | ignored | ignored | ignored |
| `running` | aborts current run | shuts down runtime after current run | → `stopping` | → `killed` | → `paused` | ignored | delivered |
| `idle` | ignored (not processing) | ignored (new code picked up on next wake) | → `stopped` | → `killed` | → `paused` | ignored | ignored (not processing) |
| `paused` | ignored (not processing) | ignored (new code picked up on next wake) | → `stopping` | → `killed` | ignored | → `running` | ignored (not processing) |
| `stopping` | ignored | ignored | ignored | → `killed` | ignored | ignored | ignored |
| `stopped` | server rejects | server rejects | server rejects | server rejects | server rejects | server rejects | server rejects |
| `killed` | server rejects | server rejects | server rejects | server rejects | server rejects | server rejects | server rejects |
## Signal Set
### Run-level signals
These affect the current run. The entity stays alive.
#### SIGINT — Interrupt Current Run
1. Server inserts `signal(SIGINT)` into the entity's StreamDB
2. Runtime effect fires — no entity state transition (entity remains `running`)
3. Runtime immediately aborts in-progress tool calls and LLM generation
4. Entity is ready to process the next message
SIGINT is the "stop generating" button.
#### SIGHUP — Reload Entity Code
1. Server inserts `signal(SIGHUP)` into the entity's StreamDB
2. Runtime effect fires — no entity state transition (entity remains `running`)
3. If entity is mid-run, the current run completes on the current code version
4. Runtime shuts down immediately after the run finishes (skips the normal idle timeout)
5. Next wake starts on the new code version
SIGHUP is the deployment signal: finish current work, shut down immediately (skip the idle timeout), pick up new code on next wake.
### Entity-level signals
These change the entity's lifecycle state — unlike run-level signals, the entity itself enters a new state.
#### SIGTERM — Graceful Entity Shutdown
1. Server inserts `signal(SIGTERM)` into the entity's StreamDB
2. Runtime effect fires, validates transition, writes `stopping` state transition (includes grace period `deadline` timestamp)
3. Runtime starts grace period timer
4. Entity code runs cleanup within the remaining grace period
5. If cleanup completes early, runtime writes `stopped` state transition, stops message delivery
6. When the grace period expires, runtime writes `stopped` state transition and stops message delivery, whether or not cleanup finished
The grace period is configurable per entity profile (default 30s, matching Kubernetes).
#### SIGKILL — Immediate Entity Shutdown
1. Server inserts `signal(SIGKILL)` into the entity's StreamDB
2. Runtime effect fires, validates transition, writes `killed` state transition
3. Runtime immediately aborts in-progress tool calls and LLM generation
4. Runtime stops message delivery
5. Entity code cannot handle this signal
#### SIGSTOP — Pause Entity
1. Server inserts `signal(SIGSTOP)` into the entity's StreamDB
2. Runtime effect fires, validates transition, writes `paused` state transition
3. Runtime stops delivering new messages to the entity
4. Messages keep arriving on the StreamDB and queue until the entity resumes
5. If entity is mid-run, the current run completes, then the runtime pauses without starting the next run
#### SIGCONT — Resume Entity
1. Server inserts `signal(SIGCONT)` into the entity's StreamDB
2. Runtime effect fires, validates transition, writes `running` state transition
3. Runtime resumes message delivery
4. Queued messages on the StreamDB are delivered normally
### User-defined signals
#### SIGUSR — User-Defined
1. Server inserts `signal(SIGUSR)` into the entity's StreamDB (with user-provided payload)
2. Runtime effect fires — no state transition, since SIGUSR doesn't change lifecycle state
3. Runtime immediately invokes developer's `onSignal` hook with the signal payload
4. The runtime does nothing else — the developer defines what SIGUSR means
## Implementation Scope
Server:
- `POST /{entity_type}/{instance_id}/signal` endpoint
- Migrate `DELETE /{entity_type}/{instance_id}` to SIGKILL
Runtime:
- TanStack DB effects for signal handling
- Signal validation and state transitions
- `onSignal` hook for developer-defined handlers
CLI:
- `electric signal {entity_type}/{instance_id} {SIGNAL}` command
App (dashboard):
- UI for sending signals to entities (stop, pause, resume, kill, interrupt)
Docs:
- Signal reference (all signal types, behaviors, state transitions)
- `onSignal` hook API
- Migration guide from DELETE to SIGKILL
Client:
- TypeScript SDK support for sending signals
## Open Questions
### `onSignal` execution context
The `onSignal` handler fires at once for SIGINT and SIGUSR, possibly while the agent is mid-run. How does the handler interact with the agent's execution context?
- Can it read the agent's conversation history / current context?
- Can it mutate state (e.g., set a flag the agent loop reads on the next step)?
- Can it write to the entity's StreamDB?
- Does it run in the same execution context as the agent loop, or a separate one?
- SIGINT aborts mid-generation, then the handler runs — does the handler see the partial output?
Contributor guide
Assessment
This issue has not been assessed yet.