Flagsmith / Flagsmith/flagsmith
Track external warehouse delivery per object to allow retries and prevent concurrent runs
- Dominant language
- Python
- Stars
- 6.6k
- Forks
- 567
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 121
Description
Delivery to external warehouses (#8107) tracks progress purely through S3 prefixes: pending objects live under `events/`, delivered ones move to `archive/`, and rejected ones to `failed/`. Two problems follow from having no per-object state.
## Transient failures cause permanent data loss
An object is moved to `failed/` on its first rejection, with no retry. A rejection code like 41 or 117 usually means bad content, but the same codes surface for reasons that would succeed on a second attempt. `failed/` is covered by the bucket's `expire-objects` lifecycle rule (prefix `""`, 30 days), so those events are deleted permanently with no operator action available.
## Concurrent runs are possible
`deliver_events_for_connection` is bounded by a time budget so it finishes inside its 9-minute task timeout, which is what currently prevents overlap. If a run does exceed the timeout, the task processor abandons the still-running thread (`future.result(timeout=...)` then shuts the executor down with `wait=False`) and retries the task, so two runs can deliver the same objects — duplicate inserts, and a race between `copy_object`/`delete_object` where the loser gets a 404.
## Proposal
Add a per-object delivery record, keyed on `(connection, s3_key)`, holding `delivered_at`, `attempts` and `last_error`. That gives:
- **Retries with a cap.** Transient failures are retried; an object is only abandoned after N attempts, so a blip no longer costs the data.
- **Claim/lease as mutual exclusion.** `UPDATE ... SET claimed_at = now() WHERE s3_key = ... AND claimed_at IS NULL` is atomic, so concurrent runs cannot take the same object — finer-grained than a per-connection lock and with no new Redis keys.
- **Bounded batches.** "Claim up to N" replaces the time budget.
- **Audit.** What was delivered, when, and why anything failed, without listing S3.
Delivery stays at-least-once: the record is written after a successful insert, so a crash in between replays the object.
Contributor guide
Research direction
Start with the deliver_events_for_connection entry point and the external warehouse delivery work referenced in #8107. Trace the current S3-prefix state transitions and timeout behavior, then define the per-object record, capped retries, atomic claims, bounded batches, and audit fields described here. Done means concurrent runs cannot claim the same object and transient failures can retry without losing delivery state.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- backend, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100