Flagsmith / Flagsmith/flagsmith

Track external warehouse delivery per object to allow retries and prevent concurrent runs

Open
#8,149 0 comments 0 reactions 0 assignees View on GitHub
api
Dominant language
Python
Stars
6.6k
Forks
567
Avg merge
1d 13h
Merged PRs (30d)
121

Description

Delivery to external warehouses (#8107) tracks progress purely through S3 prefixes: pending objects live under `events/`, delivered ones move to `archive/`, and rejected ones to `failed/`. Two problems follow from having no per-object state.

## Transient failures cause permanent data loss

An object is moved to `failed/` on its first rejection, with no retry. A rejection code like 41 or 117 usually means bad content, but the same codes surface for reasons that would succeed on a second attempt. `failed/` is covered by the bucket's `expire-objects` lifecycle rule (prefix `""`, 30 days), so those events are deleted permanently with no operator action available.

## Concurrent runs are possible

`deliver_events_for_connection` is bounded by a time budget so it finishes inside its 9-minute task timeout, which is what currently prevents overlap. If a run does exceed the timeout, the task processor abandons the still-running thread (`future.result(timeout=...)` then shuts the executor down with `wait=False`) and retries the task, so two runs can deliver the same objects — duplicate inserts, and a race between `copy_object`/`delete_object` where the loser gets a 404.

## Proposal

Add a per-object delivery record, keyed on `(connection, s3_key)`, holding `delivered_at`, `attempts` and `last_error`. That gives:

- **Retries with a cap.** Transient failures are retried; an object is only abandoned after N attempts, so a blip no longer costs the data.
- **Claim/lease as mutual exclusion.** `UPDATE ... SET claimed_at = now() WHERE s3_key = ... AND claimed_at IS NULL` is atomic, so concurrent runs cannot take the same object — finer-grained than a per-connection lock and with no new Redis keys.
- **Bounded batches.** "Claim up to N" replaces the time budget.
- **Audit.** What was delivered, when, and why anything failed, without listing S3.

Delivery stays at-least-once: the record is written after a successful insert, so a crash in between replays the object.

Contributor guide

Open the contributing guide

Research direction

Start with the deliver_events_for_connection entry point and the external warehouse delivery work referenced in #8107. Trace the current S3-prefix state transitions and timeout behavior, then define the per-object record, capped retries, atomic claims, bounded batches, and audit fields described here. Done means concurrent runs cannot claim the same object and transient failures can retry without losing delivery state.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
backend, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.