cds-snc / cds-snc/platform-forms-client

Improve Response Archiver to Handle Large Volumes Efficiently

Open
#6,193 0 comments 0 reactions 0 assignees View on GitHub
:vertical_traffic_light: Minor development
Dominant language
TypeScript
Stars
46
Forks
16
Avg merge
1d 9h
Merged PRs (30d)
91

Description

### Description:

Recent load tests have highlighted a limitation with our current response archiver implementation. When the database contains a large number of responses, the existing Lambda function may fail to process all items within its 15-minute execution window. This can result in responses that are ready to be archived remaining in the database indefinitely.

### Proposed Solution:

We should explore a new approach leveraging DynamoDB features to make archiving more reliable and scalable:

- DynamoDB TTL: Configure TTL on responses (including actual data and confirmation codes) to automatically expire items after a defined number of days.
- DynamoDB Streams: Use DynamoDB Streams to detect when a response is deleted due to TTL expiration.
- Archival Lambda: Connect a Lambda function to the stream to archive deleted responses to S3, including attachments if necessary.

Optional Enhancement:

To reduce risk, we could introduce an intermediate Lambda that pushes deleted responses to an SQS queue before archival, providing an extra layer of reliability and decoupling.

### Motivation / Benefits:

- Ensures responses are archived reliably even under heavy load.
- Removes dependency on a single long-running Lambda process.
- Leverages native DynamoDB features for efficiency and scalability.

Contributor guide

Open the contributing guide

Research direction

Start by locating the current response archiver and reviewing the load-test behavior described in the issue. Map how responses, confirmation codes, and attachments are stored and archived, then evaluate a DynamoDB TTL/Streams design, with SQS as an optional reliability layer; done means reliable archival under large volumes without the current execution-window failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
backend, cloud, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.