cds-snc / cds-snc/platform-forms-client
Improve Response Archiver to Handle Large Volumes Efficiently
- Dominant language
- TypeScript
- Stars
- 46
- Forks
- 16
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 91
Description
### Description:
Recent load tests have highlighted a limitation with our current response archiver implementation. When the database contains a large number of responses, the existing Lambda function may fail to process all items within its 15-minute execution window. This can result in responses that are ready to be archived remaining in the database indefinitely.
### Proposed Solution:
We should explore a new approach leveraging DynamoDB features to make archiving more reliable and scalable:
- DynamoDB TTL: Configure TTL on responses (including actual data and confirmation codes) to automatically expire items after a defined number of days.
- DynamoDB Streams: Use DynamoDB Streams to detect when a response is deleted due to TTL expiration.
- Archival Lambda: Connect a Lambda function to the stream to archive deleted responses to S3, including attachments if necessary.
Optional Enhancement:
To reduce risk, we could introduce an intermediate Lambda that pushes deleted responses to an SQS queue before archival, providing an extra layer of reliability and decoupling.
### Motivation / Benefits:
- Ensures responses are archived reliably even under heavy load.
- Removes dependency on a single long-running Lambda process.
- Leverages native DynamoDB features for efficiency and scalability.
Contributor guide
Research direction
Start by locating the current response archiver and reviewing the load-test behavior described in the issue. Map how responses, confirmation codes, and attachments are stored and archived, then evaluate a DynamoDB TTL/Streams design, with SQS as an optional reliability layer; done means reliable archival under large volumes without the current execution-window failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- backend, cloud, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100