Worker Queue Roadmap

## Description

This feature allows us to delegate processing of potentially long tasks to a dedicated worker process. By delegating the processing and actual completion of such tasks to another process we allow the main server to focus on handling the requests and minimise latency for users.

## Implementation

This will be implemented using the BullMQ library. It provides queue and worker management and stores the queues in Redis.
It will require the creation of a new entrypoint that defined the bullmq worker, with a connection to the database and redis.
It will also require a modification of the infrastructure to allow launching this new entrypoint inside a ECS Task with the proper resources allocated and scaling policies defined. Allowing scale-out and scale-in on appropriate parameter values will make the worker able to replicate to handle large loads and scale-in when there is not much to do. BullMQ should support concurrent workers by default.

## Planning

We will need to cary out the following tasks

- Add worker task to infrastructure with proper security groups and variable access
- Define new user credentials inside the database for secure access
- Create Dockerfile for building worker image and pushing them to ECR
- Define local development commands and infrastructure to test and dev on workers locally
- update the "/export" endpoint to dispatch to a queue
- Create the worker entrypoint allowing processing of jobs sent to the queue
- Move export logic to the new worker entrypoint, modifiying the delivery method from Blob reply to storing zip in s3 and sending an email
- Update the frontend to use the new endpoint and provide appropriate feedback to the user