Add query fallback when Electric disconnects

## Summary

Implement a fallback mechanism that allows Electric to serve shape data even when logical replication is unavailable. When the replication client is not ready, shape requests will query the database directly and return the data to clients. Clients will automatically detect fallback mode and poll for status updates, seamlessly reconnecting when replication is restored.

## Background

Electric relies on PostgreSQL's logical replication to provide real-time sync. However, several scenarios can cause replication to become unavailable:

1. **Database restarts** - Replication slot doesn't exist yet
2. **Long-running transactions** - Block replication slot creation
3. **Configuration issues** - Wrong replication settings
4. **Network partitions** - Electric can't reach database
5. **Resource constraints** - Database rejects replication connections

Currently, when replication is unavailable, Electric returns 503 errors or times out on shape requests, providing no data to clients and no graceful degradation path.

## Goals

1. **Graceful degradation** - Serve data even when replication is unavailable
2. **Automatic recovery** - Switch back to live mode when replication restores
3. **Transparent to clients** - Minimal changes to client code
4. **Clear signaling** - Clients know when they're in fallback mode
5. **Low server overhead** - CDN-cacheable status checks

## Non-Goals

1. Real-time updates in fallback mode (polling-based is acceptable)
2. Full parity with live mode performance
3. Complex client-side configuration
4. Automatic detection of *why* replication failed

## Proposed Solution

### Architecture Overview

```
┌─────────────────────────────────────────────────────────┐
│ Client (Browser/App) │
│ │
│ ┌────────────────────────────────────────────────┐ │
│ │ ShapeStream │ │
│ │ - Subscribes to shape │ │
│ │ - Detects fallback mode from header │ │
│ │ - Polls /v1/status every 10s when in fallback │ │
│ │ - Auto-reconnects when live mode detected │ │
│ └────────────────────────────────────────────────┘ │
└──────────────────────┬───────────────────────────────────┘
│
│ HTTP Requests
↓
┌─────────────────────────────────────────────────────────┐
│ CDN/Proxy │
│ - Caches /v1/status responses (5s) │
│ - Reduces load on Electric server │
└──────────────────────┬───────────────────────────────────┘
│
↓
┌─────────────────────────────────────────────────────────┐
│ Electric Server │
│ │
│ ┌────────────────────────────────────────────────┐ │
│ │ StatusMonitor │ │
│ │ - Tracks replication_client_ready condition │ │
│ │ - Returns replication_available: boolean │ │
│ └────────────────────────────────────────────────┘ │
│ │ │
│ ┌────────────────────┴──────────────────────────┐ │
│ │ API Request Handler │ │
│ │ - Checks StatusMonitor.status() │ │
│ │ - Sets fallback_mode if !replication_available│ │
│ └────────────────────┬──────────────────────────┘ │
│ │ │
│ ┌─────────────┴─────────────┐ │
│ ↓ ↓ │
│ ┌─────────────┐ ┌──────────────┐ │
│ │ Live Mode │ │ Fallback Mode│ │
│ │ - Stream │ │ - Query DB │ │
│ │ from WAL │ │ directly │ │
│ │ - Real-time │ │ - Snapshot │ │
│ │ updates │ │ format │ │
│ └─────────────┘ └──────────────┘ │
└──────────────────────┬───────────────────────────────────┘
│
↓
┌─────────────────────────────────────────────────────────┐
│ PostgreSQL Database │
│ - Logical replication (when available) │
│ - Direct queries (fallback mode) │
└─────────────────────────────────────────────────────────┘
```

## Implementation Components

### Server-Side (Elixir)

#### 1. Status Tracking (`lib/electric/status_monitor.ex`)
- Add `replication_available` boolean to status response
- Derive from existing `replication_client_ready` condition

#### 2. Fallback Detection (`lib/electric/shapes/api.ex`)
- Check replication status during request validation
- Set `fallback_mode` flag on request
- Add to response headers

#### 3. Fallback Response (`lib/electric/shapes/api.ex`)
- Query database directly using `Shapes.query_subset`
- Format as shape log with all rows as "insert" operations
- Set `up_to_date: true` to prevent unnecessary client polling
- Return 200 with `electric-fallback-mode: true` header

#### 4. Status Endpoint (`lib/electric/plug/status_plug.ex`)
- New `GET /v1/status` endpoint
- Returns: status summary ("live"/"fallback"/"starting"), replication_available, connection state, shape state
- Cache-Control: 5 seconds for CDN efficiency

### Client-Side (TypeScript)

#### 1. Fallback Detection (`src/client.ts`)
- Read `electric-fallback-mode` header from response
- Track fallback state

#### 2. Status Polling (`src/client.ts`)
- When fallback detected, start polling `/v1/status` every 10 seconds
- Check `replication_available` and `status` fields
- When "live" mode detected, trigger automatic reconnect

#### 3. Auto-Recovery
- Call `forceDisconnectAndRefresh()` when replication restored
- Seamlessly switch back to live replication mode
- Stop status polling

#### 4. Cleanup
- Stop polling on unsubscribe or reset
- Clean up timers

## Error Response Format

### Current (Replication Unavailable)
```
503 Service Unavailable
{
"message": "Timeout waiting for replication client"
}
```

### Proposed (Fallback Mode)
```
200 OK
electric-fallback-mode: true
electric-handle: shape_abc123
electric-offset: -1
electric-up-to-date: true

[
{"headers": {"operation": "insert"}, "value": {"id": 1, "name": "Item 1"}},
{"headers": {"operation": "insert"}, "value": {"id": 2, "name": "Item 2"}}
]
```

## Status Endpoint Response

```json
{
"status": "live" | "fallback" | "starting",
"replication_available": boolean,
"connection": "up" | "starting" | "waiting_on_lock" | "sleeping",
"shape": "up" | "starting"
}
```

## Example Client Usage

```typescript
const stream = new ShapeStream({
url: 'http://localhost:3000/v1/shape',
params: { table: 'items' }
})

// Automatically handles fallback mode and recovery
stream.subscribe(messages => {
// Receives data in both live and fallback modes
console.log(messages)
})
```

## Benefits

### For Clients
- **Programmatic detection**: Detect fallback mode via header
- **Automatic recovery**: No manual intervention needed
- **Continuous data**: Receive data even during replication outages

### For Operations
- **Better availability**: Data served even without replication
- **Improved UX**: Users see data instead of errors
- **Low overhead**: CDN handles most status checks

### For Support
- **Clearer status**: Know when system is degraded
- **Better monitoring**: Track fallback mode usage

## Trade-offs

### Benefits
1. **Improved availability** - Data served even without replication
2. **Better UX** - Users see data instead of errors
3. **Automatic recovery** - No manual intervention needed
4. **Low overhead** - CDN handles most status checks
5. **Simple implementation** - Reuses existing query infrastructure

### Drawbacks
1. **Not real-time** - Fallback mode is snapshot-only
2. **Increased DB load** - Every request queries DB directly
3. **Polling overhead** - 10s polling adds client-side work
4. **No live updates** - Changes not reflected until recovery
5. **Memory usage** - Full table queries on every request

### Mitigation Strategies
- **DB Load**: Response includes `up_to_date: true` to prevent unnecessary re-requests
- **Polling Overhead**: CDN caching (5s) reduces actual server requests
- **Live Updates**: Document clearly that fallback is degraded mode; automatic recovery ensures temporary state

## Team Feedback & Concerns

### Concerns Raised (balegas)

**Inconsistent experience**: Getting snapshots without live updates could create inconsistencies:
- First page has no live updates
- Second page retrieved later is more recent in time
- Progressive snapshots put increasing load on database with fallback query for each page

### Alternative Approach (msfstef) ⭐

**Key insight**: PG replication is as reliable as a PG read replica, with the exception that we currently don't do read-only mode to serve data while replication stream is inactive.

**Better approach**:
- If we know a replication slot **exists** and where we last left it, we could create new shapes (snapshots only) without replication active
- We know we can "resume" them once replication resumes
- This brings us closer to read replica behavior

**Goal**: Provide what someone would expect from a read replica, but in change stream format.

**Implications**:
- Track replication slot state persistently
- Serve snapshots that are "resumable" when replication comes back
- More sophisticated than simple fallback queries
- Better consistency guarantees

## Backwards Compatibility

✅ **Fully backwards compatible**:
- New header is optional; old clients ignore it
- Existing clients continue to work
- No breaking changes to API
- Feature activates automatically based on replication state

## Testing Requirements

- [ ] Unit tests for StatusMonitor replication_available logic
- [ ] Unit tests for API fallback mode detection
- [ ] Unit tests for response header setting
- [ ] Unit tests for client polling logic
- [ ] Integration tests for full request flow in fallback mode
- [ ] Integration tests for status endpoint responses
- [ ] Integration tests for recovery scenarios
- [ ] Manual testing: Simulate replication failure and verify fallback
- [ ] Manual testing: Simulate recovery and verify auto-reconnect
- [ ] Load testing: 1000 concurrent clients in fallback mode
- [ ] Load testing: CDN hit rate measurement
- [ ] Test inconsistency scenarios (pagination with stale first page)

## Performance Impact

### Expected Load (Example)
- 1000 concurrent users
- 10% in fallback mode (100 users)
- Status polling: 100 users × 1 request/10s = 10 req/s
- With CDN (5s cache): ~2 req/s to server

**Conclusion**: Minimal impact with CDN caching

## Files to Modify

### Server (Elixir)
- `lib/electric/status_monitor.ex` - Add replication_available field
- `lib/electric/shapes/api.ex` - Fallback detection & response
- `lib/electric/shapes/api/request.ex` - Add fallback_mode field
- `lib/electric/shapes/api/response.ex` - Add fallback_mode field & header
- `lib/electric/plug/status_plug.ex` - New status endpoint (create)
- `lib/electric/plug/router.ex` - Add /v1/status route

### Client (TypeScript)
- `src/client.ts` - Fallback detection, status polling, auto-recovery
- `src/constants.ts` - Add FALLBACK_MODE_HEADER constant

### Documentation
- `docs/rfcs/001-query-fallback-mode.md` - Detailed RFC (reference from PR #3402)

## Alternatives Considered

1. **Client-Configurable Polling** - Rejected: CDN caching makes it unnecessary
2. **Server-Initiated Reconnect** - Rejected: Requires WebSocket/SSE, added complexity
3. **Virtual System Shape** - Rejected: 40-60 hours of dev effort, overly complex
4. **No Fallback** - Rejected: Poor user experience

## Future Work

### Short Term
- Metrics: Track fallback mode usage, duration, recovery time
- Logging: Better visibility into fallback events
- Testing: Comprehensive test suite
- Address inconsistency concerns from balegas

### Medium Term
- **Implement msfstef's approach**: Track replication slot state and serve resumable snapshots
- Configurable interval: Allow server to suggest poll interval via retry-after
- Smart polling: Exponential backoff for long outages
- Admin notifications: Alert when fallback mode entered

### Long Term
- Virtual system shapes: Implement `_system/status` as queryable shape
- **Read replica behavior**: Full read-only mode that serves consistent snapshots
- Partial replication: Serve some shapes live, others fallback
- Offline mode: Extend to support offline scenarios

## Open Questions

### Design Questions
1. Should we implement msfstef's replication slot tracking approach first before the simpler fallback?
2. How do we handle the inconsistency issues raised by balegas (pagination with stale pages)?
3. Should we prevent pagination in fallback mode to avoid inconsistencies?
4. Do we need to expose replication slot state in the status endpoint?

### Operational Questions
5. Should we add metrics to track how often fallback mode is entered?
6. Should we expose fallback mode in the admin dashboard?
7. Should we add configurable polling intervals in a future iteration?
8. Should we prioritize implementing exponential backoff for status polling?

### Implementation Questions
9. Can we persist replication slot position to enable resumable snapshots?
10. What's the migration path if we want to switch from simple fallback to resumable snapshots later?

## Related

- Original PR: #3402
- RFC Document: `packages/sync-service/docs/rfcs/001-query-fallback-mode.md` (in PR #3402)
- Related to status monitoring and graceful degradation features
- Discussion with team feedback from @balegas and @msfstef

---

**Note**: This feature was originally implemented in PR #3402 including a comprehensive RFC document. The PR is being converted to an issue for further discussion and planning. **Important feedback from @msfstef suggests an alternative approach using replication slot tracking for resumable snapshots, which may be superior to the simple fallback query approach.**