aws-samples / aws-samples/sample-collaborative-ai-dlc

[Bug]: Yjs collaboration saturates a single task and cannot safely scale

Open
#455 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
JavaScript
Stars
75
Forks
23
Avg merge
3d 17h
Merged PRs (30d)
24

Description

## Description

The collaboration service can saturate as more people work concurrently. It currently runs a single ECS Fargate task with 0.5 vCPU / 1 GiB in production, and maintains each Yjs document only in that process's memory. Increasing the replica count would send collaborators to independent copies of the same document.

The DynamoDB on-demand change in #445 addresses a separate configured capacity limit. The Yjs server itself does not access DynamoDB today.

## Confirmed findings

- Task CPU, memory, and replica count are fixed; there is no capacity policy or measured collaboration workload limit.
- Document updates and awareness are not shared between tasks. Task replacement and overlapping deployments need explicit handling.
- The initial handshake sends the full document twice to a new client.
- Clients echo received awareness changes and miss unchanged-state awareness renewals. The server then attributes echoed awareness IDs to the wrong connection, allowing an unrelated disconnect to remove another user's presence.
- Broadcast queues have no application-level slow-consumer limit or resource admission policy.
- Editable CRDT state is not persisted by the server. Recovery depends on surviving browsers or business-data snapshots.
- Eligible collaborators can independently autosave the same shared content. Saves need bounded delay, retry, ordering, and reliable flush behavior.

Local reproductions confirmed independent state in two server contexts, duplicate 1 MiB initial-sync messages, and awareness ownership/cleanup errors. The original customer's deployed revision and incident metrics are still needed to attribute the production failure precisely.

## Expected behavior

Support an explicit manual capacity configuration and a safe path to horizontal scaling. Adding capacity must preserve convergence, authorization, and recoverability. Automatic scaling must not terminate a document owner before its state is recoverable.

## Implementation and validation

- [ ] Fix awareness transport, ownership, initial sync, heartbeat, and reconnect behavior.
- [ ] Add bounded connections/documents/payloads/send queues, graceful shutdown, and useful service metrics.
- [ ] Expose validated CPU/memory settings and document manual scaling.
- [ ] Add durable binary CRDT recovery and reliable persistence.
- [ ] Implement document-aware routing/ownership, or a shared-stream design if hot-document load requires it; preserve the existing public endpoint and scoped authorization.
- [ ] Verify manual scale-out/scale-in and replacement with concurrent edits before enabling automatic capacity changes.
- [ ] Add autoscaling controls with safe draining and explicit minimum/maximum capacity where supported by the selected architecture.
- [ ] Cover many-document load, a hot document, slow consumers, token-refresh/reconnect storms, worker failure, and long-running sessions.
- [ ] Verify that adding collaborators does not unnecessarily multiply backend saves.

Document the tested capacity envelope and remaining single-document limits. Do not claim customer-scale validation without a representative workload.

Contributor guide

Open the contributing guide

Research direction

No source file or test entry point is named. Start with the confirmed local reproductions for independent state, duplicate initial sync, and awareness cleanup errors, then map the collaboration service before choosing a scaling design. Done means safe scale-out and replacement with convergence, authorization, recovery, bounded resources, persistence, and a documented tested capacity envelope.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, javascript
Domain
backend, cloud, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.