electric-sql / electric-sql/electric

feat: Add structured error codes for stack unavailability

Open
#3,469 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
10.4k
Forks
375
Avg merge
3d 1h
Merged PRs (30d)
18

Description

## Summary

Add machine-readable error codes to all 503 responses when the Electric stack cannot respond to requests. This enables clients to programmatically detect stack unavailability without parsing error message strings.

## Background

Originally proposed in PR #3357, this feature adds structured error information to help clients, operations teams, and support staff better understand and handle stack unavailability scenarios.

## Proposed Changes

- **New Module**: `Electric.Shapes.Api.ErrorCode` with single `STACK_UNAVAILABLE` error code
- **Updated Response**: Add `error_code` field to Response struct
- **Updated StatusMonitor**: Return structured error information with both message and error code
- **Updated API error handling**: All stack unavailability errors should include error code and be marked as known errors
- **Tests**: Add test coverage for error code functionality

## Error Response Format

### Current
```json
{
"message": "Timeout waiting for database connection pool (snapshot) to be ready"
}
```

### Proposed
```json
{
"message": "Timeout waiting for database connection pool (snapshot) to be ready",
"code": "STACK_UNAVAILABLE"
}
```

## Design Decisions

- **Single error code**: All stack unavailability scenarios use the same `STACK_UNAVAILABLE` code. The message already provides component-specific details.
- **No retry metadata**: Retry timing is handled by the `RETRY-AFTER` header (being added in another PR). The error code is only for high-level categorization.
- **Simple and clean**: Just adds a `code` field - no additional metadata like `component`, `retryable`, or `backoff_ms`.

## Benefits

### For Clients
- **Programmatic detection**: Reliably detect stack unavailability without string parsing
- **Better error handling**: Can differentiate between stack unavailable (503 with code) vs other errors (503 without code)
- **Future-proof**: Easy to add more error codes for other scenarios (client errors, etc.)

### For Operations
- **Better monitoring**: Track stack unavailability errors separately from other 503s
- **Cleaner metrics**: Count `STACK_UNAVAILABLE` errors as a single category

### For Support
- **Clearer bug reports**: Users can provide error code
- **Consistent categorization**: All stack unavailability issues have the same code

## Example Client Usage

```typescript
async function fetchShape(url: string): Promise {
const response = await fetch(url);

if (response.status === 503) {
const error = await response.json();

if (error.code === 'STACK_UNAVAILABLE') {
// Stack is unavailable - check RETRY-AFTER header for retry timing
const retryAfter = response.headers.get('retry-after');
console.log(`Stack unavailable: ${error.message}`);
console.log(`Retry after: ${retryAfter}`);

// Implement retry logic based on RETRY-AFTER header
// ...
}
}

return response;
}
```

## Backwards Compatibility

✅ **Fully backwards compatible** - all changes should be additive:
- Error messages unchanged
- HTTP status codes unchanged
- Existing `timeout_message/1` function preserved
- Error code is an additional field

Clients that don't check for error codes will continue to work using HTTP status codes and message parsing.

## Testing Requirements

- [ ] Verify `STACK_UNAVAILABLE` code is returned in all stack 503 responses
- [ ] Verify error code field is only added when error_code is provided
- [ ] Test StatusMonitor timeout error detection
- [ ] Verify backwards compatibility (existing tests still pass)
- [ ] Manual testing of various stack unavailability scenarios

## Related

- Original PR: #3357
- Related to RETRY-AFTER header implementation (mentioned in design decisions)

## Open Questions

- Should we consider adding more specific error codes in the future for different types of stack unavailability?
- Are there other 503 scenarios besides stack unavailability that should have error codes?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.