Automattic / Automattic/wordpress-atmosphere

Feature Request: Allow TID::generate() to mint historical TIDs for backfilled posts

Open
#89 0 comments 0 reactions 1 assignee Claimed by @kraftbj View on GitHub
[Feature] Transformer enhancement
Dominant language
PHP
Stars
52
Forks
3
Avg merge
2d 1h
Merged PRs (30d)
22

Description

### What

Investigate adding an optional historical-timestamp argument (or a sibling method) to `TID::generate()` so that backfilled records can carry rkeys that reflect the original publish time rather than the time of the backfill run.

This is paired with the WP-CLI backfill request (issue 88) — the CLI is what makes large-scale historical backfills practical, and once you're backfilling thousands of posts, having every rkey clustered around "now" stops being a curiosity and starts being a real ordering problem.

### Why

Today every transformer's `get_rkey()` calls `TID::generate()`, which uses `microtime(true)`. The record's `createdAt` field is correct (it already reads `post_date_gmt` in `class-post.php:171`, `class-comment.php:85`, etc.), so Bluesky and standard.site clients display the right date. But the rkey itself encodes "now" — so a 2019 post backfilled in 2026 lives in the repo at a 2026 TID.

For anything that reads the repo in TID order (firehose consumers, repo browsers, archival tooling, ozone, anything that assumes "later TID == later content") this misorders the whole historical catalog. The transformer already knows the original publish time; the TID just doesn't use it.

### How (investigation)

A handful of things to work through before this is implementable:

1. **API shape.** Two reasonable options:
- Overload: `TID::generate( ?int $microseconds = null ): string` — null preserves the current behavior, an explicit value mints a historical TID.
- Sibling method: `TID::generate_for_time( DateTimeInterface $when ): string` — louder at call sites, doesn't need a flag to disable the monotonic floor.

I lean sibling, because the semantics of the two paths diverge enough (see point 2) that a single signature would lie about what it does.

2. **Monotonic floor.** The persisted `atmosphere_tid_last_ts` option is a forward floor for *current-time* TIDs (the CAS dance in `class-tid.php` exists specifically to keep two PHP-FPM workers from regressing it). A historical timestamp will always be below that floor, so the existing `if ( $ts <= $floor ) { $ts = $floor + 1; }` snaps it right back to "now," defeating the entire change. Historical writes have to skip the floor check entirely and — equally important — must *not* update the floor on the way out. The floor is for live publishing only.

3. **Collision within a second.** `post_date_gmt` is MySQL datetime, so resolution is seconds. Two posts published in the same second will hash to identical microseconds. That leaves only the 10-bit `$clock_id` (1024 values) to disambiguate, and within a single backfill run the `$clock_id` is fixed for the process — so two same-second posts in one batch produce identical TIDs. Options to fix:
- Spread same-second posts deterministically across microseconds by post ID (`$seconds * 1_000_000 + ( $post_id % 1_000_000 )`).
- Maintain a per-second counter for the duration of a batch.
- Catch the AT Protocol write failure and retry with `microtime(true)` — but that throws away the whole point for the retried record.

Probably worth seeding the microsecond portion from the post ID so the encoding is deterministic and re-running the backfill mints the same TID.

4. **rkey uniqueness against existing records.** AT Protocol enforces unique rkey per `(repo, collection)`. If a backdated TID happens to collide with a record already in the PDS, `applyWrites` fails. Need either a pre-check, or a catch-and-retry path.

5. **AT Protocol semantics.** The spec describes TIDs as timestamp-based but doesn't require commit-order monotonicity at the record level — historical TIDs should be legal. Worth confirming in the spec and noting any client assumptions (firehose consumers especially) that might break.

6. **Caller changes.** Five `TID::generate()` call sites need to opt in:
- `includes/transformer/class-post.php:280`
- `includes/transformer/class-document.php:202`
- `includes/transformer/class-publication.php:94`
- `includes/transformer/class-comment.php:142`
- `includes/class-publisher.php:377` (the reply rkey — probably stays as "now," since replies aren't part of the backfill flow)

7. **Tests.** New cases for the historical-time path: deterministic encoding of a known instant, no floor regression after a historical call, collision behavior for same-second post pairs. Existing monotonicity tests stay scoped to the no-argument path.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.