electric-sql / electric-sql/electric
Expand agents-server conformance coverage for Cloud/reference parity
- Dominant language
- TypeScript
- Stars
- 10.4k
- Forks
- 375
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 18
Description
## Context
We confirmed that Electric Cloud is primarily running the **agents server conformance suite** for agents-server compatibility. That means any agents-server behavior that only exists in `packages/agents-server/test/*.test.ts` is not necessarily being validated against Cloud.
Current conformance package:
- `packages/agents-server-conformance-tests`
- Main suite: `packages/agents-server-conformance-tests/src/electric-agents-tests.ts`
- Used by reference implementation via: `packages/agents-server/test/conformance.test.ts`
The reference server conformance harness currently calls:
- `runElectricAgentsConformanceTests(config)`
- `runCliConformanceTests(config)` when CLI exists
- `runMockAgentTests(config)`
- `runMockAgentCliTests(config)` when CLI exists
The conformance suite already covers a lot of core public entity runtime behavior, but a large amount of important agents-server behavior remains only in reference implementation tests under `packages/agents-server/test`.
We should expand the conformance suite so Cloud and the reference implementation are tested against the same externally visible contract.
## Current conformance coverage
The conformance suite already covers many core public behaviors:
### Entity lifecycle
- spawn registered entity type
- reject unknown entity type
- reject duplicate entity URL
- spawn without explicit body type
- get/list/ps
- status filters
- kill
- send after kill rejected
- stream persists after kill
- full spawn → send → webhook → read → kill lifecycle
### Entity type registry
- register/list/inspect/delete entity type
- duplicate registration upserts
- reject malformed registration
- minimal type registration without `creation_schema`
### Typed spawn / schema validation
- validate `creation_schema`
- reject invalid spawn args
- parent spawn handling
- explicit/default tags at spawn
- reject non-string tags at spawn
### Send/schema validation
- validate `input_schemas`
- reject invalid/unknown typed messages
- behavior with no schemas / empty schemas
### Schema evolution
- schema amendment adding message types
- reject modifications to existing schema keys
- multiple schema amendments against existing entities
- amendment on deleted type
### Serve endpoint
- register type via serve endpoint
- reject serve endpoint name mismatch
### State Protocol send format
- inbox event shape
- `type`/`key`/`value`/`headers` contract
- ordering expectations
- parseability by runtime webhook handler expectations
### Error paths
- kill missing entity
- send missing payload
- delete missing type
- stopped tag write rejected without claim
### Auth boundary
- direct writes to entity stream require claim/write token
- wrong/no token rejected
- spawn/get do not leak write tokens
- unauthenticated `send` remains allowed
- non-entity stream writes remain unauthenticated
### Concurrency / property-based
- randomized action sequences preserve safety invariants
- concurrent spawns under same type
- sequential tag writes through claim token
## Problem
The conformance suite does **not** yet cover several important externally visible agents-server behaviors that are currently tested only by the reference implementation.
The reference server has many additional test files under:
```txt
packages/agents-server/test
```
Several of these tests validate public API or protocol behavior that Cloud should also satisfy.
Important gaps include:
1. claim-scoped write-token lifecycle
2. runner API and pull-wake behavior
3. dispatch policy routing
4. scheduling APIs
5. wake/manifest semantics
6. fork behavior
7. multi-tenant / owner-scoped isolation
8. shared-state / Durable Streams proxy behavior
## Goal
Move or port the externally visible, black-box agents-server behavior tests into `packages/agents-server-conformance-tests`, so they can run against:
- the reference implementation
- Electric Cloud
- any future compatible agents-server implementation
The conformance suite should define the public compatibility contract. Reference-only implementation details should stay in `packages/agents-server/test`.
## Non-goals
Do **not** move implementation-specific unit tests into conformance.
Examples that should generally stay reference-only:
- migration folder resolution
- environment variable parsing
- low-level route hook composition
- schema middleware unit tests
- URL helper unit tests
- Postgres-specific scheduler internals
- LMDB/local token store internals
- outbox retry/dead-letter implementation details
- server startup cleanup internals
- OSS dashboard/router wrapper behavior
- low-level adapter mechanics unless exposed as public protocol behavior
## Proposed structure
Split the conformance package into capability-based suites, instead of one very large catch-all runner.
Suggested exports:
```ts
runCoreEntityConformanceTests(config)
runClaimWriteTokenConformanceTests(config)
runRunnerConformanceTests(config)
runScheduleConformanceTests(config)
runWakeConformanceTests(config)
runForkConformanceTests(config)
runCliConformanceTests(config)
runMockAgentTests(config)
runMockAgentCliTests(config)
```
`runElectricAgentsConformanceTests(config)` can continue to exist as a broad/default suite, but Cloud should be able to opt into capability-specific suites explicitly.
Example:
```ts
runCoreEntityConformanceTests(config)
runClaimWriteTokenConformanceTests(config)
runRunnerConformanceTests(config)
runScheduleConformanceTests(config)
runWakeConformanceTests(config)
```
This also makes it easier to skip suites for capabilities Cloud does not yet expose.
## Suggested priority
### P0: Claim-scoped write-token lifecycle
Source/reference tests to inspect:
- `packages/agents-server/test/server-claim-write-token.test.ts`
- `packages/agents-server/test/claim-write-token-store.test.ts`
Port the externally visible behavior into conformance.
Candidate conformance scenarios:
- claim returns a scoped write token
- active claim token can write to entity stream
- no token is rejected
- wrong token is rejected
- stale token is rejected after a newer claim
- stale `done` cannot mark a newer active claim idle
- kill clears/invalidates active claim token
- tag writes accept the active claim token
- claim token is accepted in supported headers, e.g. `Authorization` / `electric-claim-token`
- claim-scoped writes validate output/state schemas
- unknown event types are rejected when output schemas exist
- arbitrary events are accepted when no state schemas exist
- writes to stopped entities are rejected even with a token
- claim-scoped tag writes reject non-string values
- claim-scoped tag writes support merge/delete semantics
Why P0:
- This is security-sensitive.
- It is core to the agent wake/write contract.
- Cloud/reference divergence here would be serious.
### P0: Runner API / pull-wake contract
Source/reference tests to inspect:
- `packages/agents-server/test/runners-router.test.ts`
- `packages/agents-server/test/pull-wake-subscription-stack.test.ts`
- `packages/agents-server/test/dispatch-policy-routing.test.ts`
- `packages/agents-server/test/webhook-forward-routing.test.ts`
- `packages/agents-server/test/runtime-registry.test.ts`
Port public API/protocol behavior into conformance.
Candidate conformance scenarios:
- register a runner
- list runners for authenticated owner
- reject unauthenticated runner listing/claims
- reject runner registration for a different owner
- canonicalize owner principal inputs
- runner health includes expected public diagnostics
- runner diagnostics are sanitized
- runner is unhealthy when lease expires
- runner is degraded/unhealthy when stream is disconnected/disabled
- spawn entity with runner-targeted dispatch policy
- send creates a runner wake
- runner claim returns enriched notification
- claim conflicts map to 409
- callback/done acknowledges wake
- pull-wake subscription receives pre-existing wake events
- missing runner dispatch subscription is recreated before send
- runner dispatch subscription creation conflicts are treated idempotently
Why P0:
- This is likely a major Cloud/reference parity surface.
- Runners are a core part of the agents architecture.
- The current conformance suite mostly exercises webhook-style wake handling, not runner pull-wake behavior.
### P1: Scheduling APIs
Source/reference tests to inspect:
- `packages/agents-server/test/electric-agents-routes.test.ts`
- `packages/agents-server/test/scheduler-integration.test.ts`
- `packages/agents-server/test/scheduler.test.ts`
- `packages/agents-server/test/server-start.test.ts`
Port only public behavior. Avoid Postgres/LISTEN/NOTIFY internals.
Candidate conformance scenarios:
- future/delayed send schedule can be created
- delayed send lands exactly once
- delayed send can be replaced
- delayed send can be cancelled/deleted
- cron schedule can be created
- cron schedule rejects invalid/missing expression
- cron schedule emits at least one expected send/wake
- deleting cron schedule prevents future delivery
- scheduled payload validation happens before enqueueing
Why P1:
- Scheduling is externally visible.
- It is easy for Cloud and reference implementation to diverge due to infrastructure differences.
### P1: Wake / manifest semantics
Source/reference tests to inspect:
- `packages/agents-server/test/wake-registry.test.ts`
- `packages/agents-server/test/wake-registry-sync.test.ts`
- `packages/agents-server/test/manifest-side-effects.test.ts`
- `packages/agents-server/test/horton-spawn-worker.test.ts`
- `packages/agents-server/test/horton-title-generation.test.ts`
- `packages/agents-server/test/horton-pull-wake-e2e.test.ts`
Candidate conformance scenarios:
- spawn with wake registers a condition
- child `runFinished` wake is delivered to parent/subscriber
- `runFinished` wake includes child text response by default
- `includeResponse: false` omits response
- failed child run includes error
- unrelated/unscoped errors are ignored
- event append triggers collection/change wake
- wake event contains source/change details
- timeout wake is delivered
- debounce coalesces rapid events
- debounced `runFinished` preserves requested status/includeResponse options
Why P1:
- Wake semantics are core to agent composition.
- Some of this is currently deeply tested in the reference server but not necessarily in Cloud.
### P1: Fork behavior
Source/reference tests to inspect:
- `packages/agents-server/test/electric-agents-routes.test.ts`
- `packages/agents-server/test/electric-agents-status.test.ts`
- `packages/agents-server/test/stream-client-fork.test.ts`
Candidate conformance scenarios:
- fork endpoint returns public entity/entities
- forked streams preserve source history
- JSON appends to forked streams are accepted
- fork is rejected or waits while subtree is active, depending on spec
- sends to subtree entities are rejected while fork snapshotting is in progress
- fork locks are released after failure
Why P1:
- Forking is visible API behavior if exposed in Cloud.
- It has concurrency/consistency semantics that should be tested across implementations.
### P2: Multi-tenant / owner isolation
Source/reference tests to inspect:
- `packages/agents-server/test/entity-projector.test.ts`
- `packages/agents-server/test/entity-type-registry.test.ts`
- `packages/agents-server/test/wake-registry.test.ts`
- `packages/agents-server/test/tag-stream-outbox-registry.test.ts`
- `packages/agents-server/test/tag-stream-outbox-drainer.test.ts`
- `packages/agents-server/test/host.test.ts`
- `packages/agents-server/test/server-utils.test.ts`
This likely requires extending the conformance DSL/config to support auth headers, tenant IDs, or principal injection.
Candidate conformance scenarios:
- tenant A/B entity type names do not collide
- tenant A cannot list tenant B entities
- tenant A cannot inspect tenant B entity type
- runner listing is scoped to authenticated owner
- runner runtime diagnostics shapes are owner-scoped
- no cross-tenant wake delivery
- tenant-specific entity streams are isolated
Why P2:
- Important for Cloud.
- May require DSL/auth harness changes before tests can be written cleanly.
### P2: Shared-state / Durable Streams proxy behavior
Source/reference tests to inspect:
- `packages/agents-server/test/electric-agents-routes.test.ts`
- `packages/agents-server/test/stream-client.test.ts`
Candidate conformance scenarios, if part of the public contract:
- shared-state stream traffic routes correctly
- subscription control-plane traffic routes correctly
- reserved `__ds` paths behave as expected
- configured bearer auth is forwarded for service-scoped subscription traffic
- subscription-scoped bearer auth is preserved for ack/proxy traffic
- subscription stream paths are relative to the resolved stream root
Some of these may be adapter-specific. Only move the externally visible compatibility requirements.
## Cleanup: remove runtime tool tests from server conformance
`runElectricAgentsConformanceTests` currently includes a block named:
```ts
Electric Agents - Tool Tests
```
These tests import directly from:
```ts
../../agents-runtime/src/tools
```
They test things like:
- bash tool stdout/stderr capture
- bash timeout
- read_file path/binary/size guard
- web_search/fetch_url tool interface
These are not really **agents-server conformance** tests. They depend on monorepo source layout and runtime implementation details.
Suggested change:
- Move these tests back to `packages/agents-runtime/test`, or
- create a separate `runRuntimeToolConformanceTests` export if we intentionally want runtime/tool conformance.
They should not run as part of black-box agents-server compatibility against Cloud.
## Implementation plan
### Step 1: Add conformance suite structure
Create additional files under `packages/agents-server-conformance-tests/src`, for example:
```txt
src/core-entity-tests.ts
src/claim-write-token-tests.ts
src/runner-tests.ts
src/schedule-tests.ts
src/wake-tests.ts
src/fork-tests.ts
```
Update `src/index.ts` to export the new suite runners.
Keep backward compatibility:
```ts
export function runElectricAgentsConformanceTests(config) {
runCoreEntityConformanceTests(config)
// optionally include currently existing broad set, or keep as alias during migration
}
```
### Step 2: Extend the DSL/config as needed
The current DSL in:
```txt
packages/agents-server-conformance-tests/src/electric-agents-dsl.ts
```
may need support for:
- default request headers
- auth/principal headers
- tenant IDs
- runner registration helpers
- claim helpers
- schedule helpers
- wait/poll helpers
- stream assertion helpers
Suggested config shape:
```ts
export interface ElectricAgentsTestOptions {
baseUrl: string
headers?: Record | (() => Record | Promise>)
capabilities?: {
runners?: boolean
schedules?: boolean
wakes?: boolean
fork?: boolean
multiTenant?: boolean
}
}
```
Need to be careful not to make tests silently skip too much. Capability skips should be explicit and visible.
### Step 3: Port P0 claim-token tests
Port from reference tests to black-box HTTP scenarios.
Avoid importing reference server internals.
### Step 4: Port P0 runner/pull-wake tests
Port public runner API and wake claiming behavior.
Avoid asserting internal database rows or implementation-specific subscription IDs unless they are part of the public response contract.
### Step 5: Port P1 schedule/wake/fork tests
Move in batches. Each batch should be runnable against both:
- local reference server
- Cloud target
### Step 6: Wire reference implementation to run all conformance suites
Update:
```txt
packages/agents-server/test/conformance.test.ts
```
to call the new suite exports.
### Step 7: Wire Cloud test job to run all supported conformance suites
Wherever Cloud currently runs conformance, update it to include the newly exported suites.
Cloud should report which conformance capabilities were run/skipped.
## Acceptance criteria
- [ ] Conformance package exports separate suite runners for core, claim-token, runner, schedule, wake, and fork behavior.
- [ ] Reference server conformance harness runs the new suites.
- [ ] Cloud conformance job runs all supported suites.
- [ ] Claim-scoped write-token lifecycle is covered in conformance.
- [ ] Runner registration/list/claim/health and pull-wake delivery are covered in conformance.
- [ ] Public scheduling API behavior is covered in conformance, if supported by Cloud.
- [ ] Public wake/manifest behavior is covered in conformance, if supported by Cloud.
- [ ] Fork behavior is covered in conformance, if supported by Cloud.
- [ ] Runtime tool tests are removed from the agents-server conformance runner or split into a separate runtime/tool suite.
- [ ] Implementation-specific tests remain in `packages/agents-server/test` and are not moved to conformance.
- [ ] Conformance tests do not import reference server internals.
- [ ] Conformance tests can run against an arbitrary `baseUrl`.
- [ ] Capability skips are explicit and visible in test output.
## Notes
The highest-value work is probably:
1. claim-scoped write-token conformance
2. runner/pull-wake conformance
These are security/correctness-critical and most likely to reveal Cloud/reference divergence.
Contributor guide
Assessment
This issue has not been assessed yet.