electric-sql / electric-sql/electric

Implement backwards-compatible transition from 409 to 200 status code for shape expiry

Open
#3,150 4 comments 1 reaction 0 assignees View on GitHub
Dominant language
TypeScript
Stars
10.4k
Forks
375
Avg merge
3d 1h
Merged PRs (30d)
18

Description

## Problem

Currently, when a shape expires and needs to be re-fetched, the server returns a 409 (Conflict) status code with a \"must-refetch\" control message. We've received feedback that users interpret 409 as an error condition, which creates confusion. Since this is a normal part of the shape synchronization protocol, not an error, we should use a 2xx success status code.

## Solution

Implement a feature negotiation mechanism using an \"accepts\" query parameter that allows clients to opt into receiving 200 instead of 409 for must-refetch scenarios, while maintaining backwards compatibility with existing clients.

To allow for observability of must-refetch control message, we'll add a new response header `electric-control` with a `must-refetch` value in this case. We should also look at using this generally e.g. right now we have a `electric-up-to-date` header which could could fit into this scheme as well. We have three control messages atm, `must-refetch`, `up-to-date`, and `snapshot-end`.

## Implementation Details

### 1. Server-side changes (Elixir)

**a) Update parameter validation** in `/packages/sync-service/lib/electric/shapes/api/params.ex`:
- Add \`accepts\` field to the embedded schema
- Parse accepts as a comma-separated list of feature strings
- Store parsed features in the request context

**b) Update error response generation** in `/packages/sync-service/lib/electric/shapes/api/error.ex`:
- Modify \`must_refetch/1\` to check if client accepts \"reset-200\" feature
- Return status 200 w/ `electric-control` header
- Else return 409 with must-refetch control message for backwards compatibility
- Always include the \`electric-handle\` header with the new shape handle

**c) Update API endpoint handlers**:
- Pass the accepts features through the request pipeline
- Ensure feature negotiation is available when generating shape expiry responses
- Set \`Content-Length: 0\` for 200 responses per RFC 7231

### 2. Client-side changes (TypeScript)

**a) Update ShapeStream options** in `/packages/typescript-client/src/client.ts`:
- Add \`accepts\` parameter to ShapeStreamOptions interface
- Default to \`[\"reset-200\"]\` for new client versions
- Allow override for testing backwards compatibility

**b) Update request URL building**:
- Add accepts parameter to query string when present
- Format as comma-separated list (e.g., \`accepts=reset-200\` or \`accepts=reset-200,other-feature\`)
- Add \"accepts\" to ELECTRIC_PROTOCOL_QUERY_PARAMS for proxy forwarding

**c) Update response handling**:
- Handle both 409 and 200 status codes for shape expiry
- For 200: Read the control message `must-refetch` like now and the `electric-handle` header and reset the shape
- For 409: Continue existing behavior with must-refetch control message
- Same reset behavior for both status codes

### 3. Update OpenAPI specification

Update `/website/electric-api.yaml`:
- Add \`accepts\` query parameter documentation
- Document 200 response for shape endpoint when shape expires
- Keep 409 response documentation for backwards compatibility
- Explain feature negotiation mechanism

## Migration Strategy

1. **Phase 1**: Deploy server support for \`accepts\` parameter (returns 200 when \`reset-200\` requested, 409 otherwise)
2. **Phase 2**: Update TypeScript client to send \`accepts=reset-200\` by default
3. **Phase 3**: Update other clients (Elixir, community clients) to adopt the new behavior
4. **Phase 4**: After sufficient adoption, consider making 200 the default for clients that don't specify accepts

## Testing Requirements

- Server returns 409 with must-refetch when accepts parameter is absent
- Server returns 409 with must-refetch when accepts doesn't include \"reset-200\"
- Server returns 200 with no body when accepts includes \"reset-200\"
- Server always includes \`electric-handle\` header on shape expiry (both 409 and 200)
- TypeScript client handles both 409 and 200 correctly for shape expiry
- Existing clients continue to work without changes
- Proxy configurations forward accepts parameter correctly

## Documentation Updates

- Update HTTP API documentation to explain the accepts parameter
- Document that shape expiry is a normal protocol operation, not an error
- Update client documentation with new accepts option
- Explain why 200 is semantically appropriate for this use case

## Implementation Example

**Client request with new behavior:**
\`\`\`
GET /v1/shape?table=users&offset=10_5&handle=abc123&accepts=reset-200
\`\`\`

**Server response when shape expired (new behavior):**
\`\`\`
HTTP/1.1 200
electric-handle: new-handle-456
electric-control: must-refetch
Content-Type: application/json

[{\"headers\": {\"control\": \"must-refetch\"}}]
\`\`\`

**Server response when shape expired (legacy behavior):**
\`\`\`
HTTP/1.1 409 Conflict
electric-handle: new-handle-456
Content-Type: application/json

[{\"headers\": {\"control\": \"must-refetch\"}}]
\`\`\`

## Benefits
1. **Backwards compatible**: Existing clients continue to work unchanged
2. **Future-proof**: Establishes a pattern for protocol evolution
3. **No confusion**: 2xx status clearly indicates expected protocol behavior, not an error

## References

- Current 409 implementation: \`/packages/sync-service/lib/electric/shapes/api/error.ex:20\`
- TypeScript 409 handling: \`/packages/typescript-client/src/client.ts\` (line with 409 check)
- OpenAPI spec 409 response: \`/website/electric-api.yaml\` (409 response definition)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.