Kinto / Kinto/kinto

bump_timestamp() trigger causes IntegrityError under concurrent writes — switch to microsecond-precision bumps

Open
#3,673 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
4.4k
Forks
437
Avg merge
1d 2h
Merged PRs (30d)
15

Description

## Problem

The `bump_timestamp()` trigger computes the next `last_modified` via millisecond-precision epoch arithmetic:

```sql
previous := as_epoch(MAX(last_modified)); -- milliseconds
current := as_epoch(clock_timestamp()::TIMESTAMP); -- milliseconds
IF previous IS NOT NULL AND previous >= current THEN
current := previous + 1; -- +1 millisecond
END IF;
NEW.last_modified := from_epoch(current);
```

This `SELECT MAX(...)` is not serialized across concurrent transactions. When two transactions insert/update rows with the same `(parent_id, resource_name)` within the same millisecond, both read the same `MAX(last_modified)`, compute the same `previous + 1`, and one fails:

```
IntegrityError: duplicate key value violates unique constraint
"idx_objects_parent_id_resource_name_last_modified"
DETAIL: Key (parent_id, resource_name, last_modified)=(...) already exists.
```

The trigger's own comment acknowledges this:

> If a bunch of requests from the same user on the same resource arrive in the same millisecond, the unicity constraint can raise an error (operation is cancelled).

In high-throughput deployments (multiple application pods writing to the same collection concurrently), this is not a rare edge case — it happens routinely under normal load.

## Proposal: microsecond-precision bumps (no API change)

The `TIMESTAMP` column already stores microsecond precision natively in PostgreSQL. The collision window is artificially wide because the trigger round-trips through millisecond-truncated epoch integers.

The fix is to work in `TIMESTAMP` directly and bump by 1 microsecond instead of 1 millisecond — reducing the collision window by 1000x without changing any client-facing behavior:

```sql
CREATE OR REPLACE FUNCTION bump_timestamp()
RETURNS trigger AS $$
DECLARE
previous_ts TIMESTAMP;
current_ts TIMESTAMP;
BEGIN
previous_ts := NULL;
WITH existing_timestamps AS (
(
SELECT last_modified
FROM objects
WHERE parent_id = NEW.parent_id
AND resource_name = NEW.resource_name
ORDER BY last_modified DESC
LIMIT 1
)
UNION
(
SELECT last_modified
FROM timestamps
WHERE parent_id = NEW.parent_id
AND resource_name = NEW.resource_name
)
)
SELECT MAX(last_modified) INTO previous_ts
FROM existing_timestamps;

current_ts := clock_timestamp()::TIMESTAMP;
IF previous_ts IS NOT NULL AND previous_ts >= current_ts THEN
current_ts := previous_ts + INTERVAL '1 microsecond';
END IF;

IF NEW.last_modified IS NULL OR
(previous_ts IS NOT NULL AND NEW.last_modified = previous_ts) THEN
NEW.last_modified := current_ts;
END IF;

RETURN NEW;
END;
$$ LANGUAGE plpgsql;
```

### Why this is backward compatible

- **`TIMESTAMP` column**: Already stores microsecond precision — no schema change needed.
- **Unique index** (`idx_objects_parent_id_resource_name_last_modified`): Operates on the `TIMESTAMP` column directly — benefits from the finer granularity automatically.
- **`as_epoch()` / HTTP API**: Continues to return millisecond-precision integers for `ETag`, `last_modified` in JSON, and `_since` filtering. No client-visible change. Two records that would have collided now get distinct microsecond timestamps but may share the same millisecond epoch — this is fine since `_since` polling is idempotent.
- **No migration needed** beyond replacing the trigger function (which is `CREATE OR REPLACE`).

### Impact

This affects any deployment where multiple processes write objects to the same `(parent_id, resource_name)` concurrently — e.g., multiple API pods behind a load balancer, concurrent webhook callbacks, or batch imports. The current workaround is application-level retry with `transaction.abort()`, which works but adds complexity that belongs in the storage layer.

Contributor guide

Open the contributing guide

Research direction

Locate the PostgreSQL definition of bump_timestamp() and its existing as_epoch()/from_epoch() helpers. Review the trigger and unique-index behavior under concurrent writes, then add or update coverage for concurrent writes with the same parent and resource. Done means microsecond TIMESTAMP bumps avoid duplicate-key failures while preserving millisecond API, ETag, and _since behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, sql
Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.