dumpus-app / dumpus-app/dumpus-api
Cost optimization: route small packages back to Lambda, big ones to Fargate
- Dominant language
- Python
- Stars
- 5
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
After PR #74 lands, every package — even tiny test ones — pays the Fargate cold-start (~30s task spin-up) and per-second pricing. For 4 packages/day that's still ~\$1-2/month, but if usage grows or skews toward a long tail of small packages, the always-Fargate path is the wrong default.
## Proposal
Before enqueueing, the API does a HEAD request on the Discord download URL to read \`Content-Length\`. Based on the size:
- **< threshold (e.g. 100 MB compressed)** → enqueue to a \"small\" SQS queue, consumed by a worker Lambda. Sub-second cold start, finishes well under the 15-min cap, cheap.
- **≥ threshold** → enqueue to a \"big\" SQS queue, consumed by the forwarder Lambda → Fargate task (the path PR #74 builds).
If the HEAD fails (Discord doesn't always return Content-Length on these URLs), fall back to the Fargate path — safer, only marginally more expensive.
## What this needs
- A 'small' worker Lambda built from the same image as the API (handler = \`lambda_handlers.worker.handler\`, which would need to be re-added — basically the file PR #74 just deleted).
- A second SQS queue + redrive policy.
- Routing logic in \`enqueue.py\` (do the HEAD, pick the queue).
- Threshold as a TF variable so we can tune it without code changes.
## When to do this
Only worth it if the cost line item starts mattering. At current volume (3-4 packages/day) the math is rounding error. Revisit when:
- Monthly Fargate cost exceeds ~\$10, **or**
- We're routinely processing > 50 packages/day and most are small, **or**
- We add a free tier with high-volume / low-size traffic.
Not urgent. Filing so we don't forget the option exists.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.