n0-computer / n0-computer/iroh-docs
`MAX_TIMESTAMP_FUTURE_SHIFT` has no safe value: a writer 10 min 1 s ahead is silently refused everywhere, a writer at 9 min 59 s holds any key for as long as it keeps stamping
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 74
- Forks
- 16
- PR merge metrics
- No merged PRs in 30d
Description
Summary. Conflicts are resolved by the writer's own timestamp
(Ord for Record, sync.rs:1158), gated only by a fixed 10-minute future bound
against the receiver's clock (sync.rs:640). I ran the two sides of that bound
through Replica::insert_remote_entry on main@8cfeacb. Both failure modes are
quiet, and they move in opposite directions with the same constant, so no value fixes
both.
Honest writer ahead. Accepted up to exactly +10 min 0 s; refused from +10 min 1 s
with TooFarInTheFuture. On the gossip path that refusal is
debug!("ignoring entry received via gossip") (engine/gossip.rs:181). The writer's
own replica accepted the same entry, so it sees a value its peers do not.
Honest writer whose clock gets corrected. One write while +1 min ahead, then
correct the clock: every subsequent write to that key is NewerEntryExists for one
minute. At +9 min 59 s, ten minutes. Fixing the clock is what locks you out.
Writer who lies inside the bound. Stamp now+9 min 59 s: accepted. Honest peers
keep writing with correct clocks: every insert returns Ok, and
single_latest_per_key never shows them. Two seconds later the liar stamps
now+9 min 59 s again: accepted. The hold never has to expire.
| bound | honest writer over it | writer just under it |
|---|---|---|
| smaller | refused sooner | wins for less |
| larger | refused later | wins for longer |
Controls. Equal clocks → later write wins. A liar with an honest stamp → loses to
a later honest write. Equal stamps → same winner whichever arrives first (the
content-digest tie-break is order-independent). So the stamp is the only input a peer
controls, and it is the whole decision.
What I am not claiming. Willow says "we are cognisant of their limitations, and
use them anyway," so the design is deliberate; I am not reporting a surprise. What I
could not find written down — in Willow, here, or the docs page that describes this as
built on CRDTs — is that the bound cannot be set safely, or that the honest-side
failure is silent. #97 fixed the unit slip that made the bound 600 ms; this is about
the 10 minutes it was meant to be. Not exercised: RBSR sessions and gossip transport
(both call insert_remote_entry); a receiver whose clock is slow (validate_entry
takes its own system_time_now(), so I could reason about it but not run it).
Reproduction. ~150 lines against the crate's public API, Docker, no
reimplementation: below. Every row above is in the
output.
If this is a known and accepted trade-off, a sentence saying so on the kv-crdts page
src/main.rs (the whole witness)
//! Witness: what iroh-docs does with a writer-declared timestamp.
//!
//! Everything below calls iroh-docs' own `Replica` API on its own in-memory
//! store, through the same `insert_remote_entry` path the gossip and sync
//! layers use. No reimplementation. Timestamps are chosen explicitly so the
//! writer's clock can be anything; the receiver's clock is the real one.
//!
//! Scope, stated narrowly: the replica insert/validate path and the
//! `single_latest_per_key` read. Not exercised: the network transport, the
//! range-based reconciliation session, or the blob layer.
use std::time::{SystemTime, UNIX_EPOCH};
use anyhow::Result;
use iroh_docs::{
store::{Query, Store},
sync::{ContentStatus, InsertError, Record, SignedEntry, ValidationFailure},
Author, NamespaceSecret, MAX_TIMESTAMP_FUTURE_SHIFT,
};
const US: u64 = 1_000_000; // microseconds per second, the unit iroh-docs uses
fn now_us() -> u64 {
SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_micros() as u64
}
fn fmt_dur(us: i128) -> String {
let s = us / US as i128;
if s.abs() >= 3600 { format!("{:+}h", s / 3600) }
else if s.abs() >= 60 { format!("{:+}m{}s", s / 60, (s % 60).abs()) }
else { format!("{:+}s", s) }
}
fn outcome(r: &std::result::Result<usize, InsertError>) -> &'static str {
match r {
Ok(_) => "accepted",
Err(InsertError::Validation(ValidationFailure::TooFarInTheFuture)) => "REJECTED TooFarInTheFuture",
Err(InsertError::NewerEntryExists) => "REJECTED NewerEntryExists",
Err(_) => "REJECTED other",
}
}
struct Net {
store: Store,
ns: NamespaceSecret,
peer: [u8; 32],
}
impl Net {
fn new() -> Self {
let mut rng = rand::rng();
Net { store: Store::memory(), ns: NamespaceSecret::new(&mut rng), peer: [7u8; 32] }
}
fn author(&self) -> Author { Author::new(&mut rand::rng()) }
/// A remote writer's entry arriving at this receiver, with the timestamp
/// the writer put on it.
async fn arrive(&mut self, author: &Author, key: &[u8], data: &[u8], ts: u64)
-> std::result::Result<usize, InsertError>
{
let rec = Record::new(iroh_blobs::Hash::new(data), data.len() as u64, ts);
let e = SignedEntry::from_parts(&self.ns, author, key, rec);
let mut replica = self.store.new_replica(self.ns.clone()).unwrap();
replica.insert_remote_entry(e, self.peer, ContentStatus::Complete).await
}
/// What a reader of this replica sees as *the* value for `key`.
fn winner(&mut self, key: &[u8]) -> Option<(String, u64)> {
let q = Query::single_latest_per_key().key_exact(key);
let mut it = self.store.get_many(self.ns.id(), q).unwrap();
it.next().map(|e| { let e = e.unwrap(); (label(&e), e.timestamp()) })
}
}
fn label(e: &SignedEntry) -> String {
format!("{}", e.author().fmt_short())
}
#[tokio::main]
async fn main() -> Result<()> {
let shift = MAX_TIMESTAMP_FUTURE_SHIFT;
println!("iroh-docs main@8cfeacb MAX_TIMESTAMP_FUTURE_SHIFT = {} us = {} s", shift, shift / US);
println!();
// ── Control A: honest writers, equal clocks ─────────────────────────
{
let mut n = Net::new();
let (a, b) = (n.author(), n.author());
let t = now_us();
let ra = n.arrive(&a, b"k", b"A", t).await;
let rb = n.arrive(&b, b"k", b"B", t + 1).await;
let w = n.winner(b"k").unwrap();
println!("CONTROL A equal clocks: A@t {}, B@t+1us {} -> reader sees {} {}",
outcome(&ra), outcome(&rb), w.0,
if w.0 == label_of(&b) { "(B, the later write) OK" } else { "UNEXPECTED" });
}
// ── Table 1: an honest writer whose clock is ahead by Δ ─────────────
println!();
println!("TABLE 1 honest writer, clock ahead by Δ, receiver clock correct");
println!("| Δ | insert result |");
println!("|---|---|");
for &d in &[1 * US, 60 * US, shift - 1 * US, shift, shift + 1 * US, 3600 * US, 86400 * US] {
let mut n = Net::new();
let a = n.author();
let r = n.arrive(&a, b"k", b"x", now_us() + d).await;
println!("| {} | {} |", fmt_dur(d as i128), outcome(&r));
}
// ── Table 2: honest writer's clock was ahead, then got corrected ───
println!();
println!("TABLE 2 same writer: one write while clock was ahead by Δ, then clock corrected, writes again");
println!("| Δ | write while ahead | write after correction |");
println!("|---|---|---|");
for &d in &[60 * US, shift - 1 * US, 3600 * US] {
let mut n = Net::new();
let a = n.author();
let r1 = n.arrive(&a, b"k", b"while-ahead", now_us() + d).await;
let r2 = n.arrive(&a, b"k", b"after-fix", now_us()).await;
println!("| {} | {} | {} |", fmt_dur(d as i128), outcome(&r1), outcome(&r2));
}
println!("(NewerEntryExists means the writer's own corrected-clock writes are refused until real time passes its earlier stamp — a self-inflicted lockout of length Δ.)");
// ── Table 3: a liar inside the bound ────────────────────────────────
println!();
println!("TABLE 3 liar stamps now+(bound-1s); honest writers keep writing with correct clocks");
let mut n = Net::new();
let (h, l) = (n.author(), n.author());
let t0 = now_us();
let who = |n: &mut Net, l: &Author| { let w = n.winner(b"k").unwrap().0; if w == label_of(l) { "LIAR" } else { "honest" } };
let r_h1 = n.arrive(&h, b"k", b"honest-1", t0).await; let s1 = who(&mut n, &l);
let r_l = n.arrive(&l, b"k", b"liar", t0 + shift - 1 * US).await; let s2 = who(&mut n, &l);
let r_h2 = n.arrive(&h, b"k", b"honest-2", t0 + 1 * US).await; let s3 = who(&mut n, &l);
let r_h3 = n.arrive(&h, b"k", b"honest-3", t0 + 1_500_000).await; let s4 = who(&mut n, &l);
println!("| step | insert result | reader sees |");
println!("|---|---|---|");
println!("| honest writes @t0 | {} | {} |", outcome(&r_h1), s1);
println!("| liar writes @t0+{} | {} | {} |", fmt_dur((shift - US) as i128), outcome(&r_l), s2);
println!("| honest writes @t0+1s | {} | {} |", outcome(&r_h2), s3);
println!("| honest writes @t0+1.5s | {} | {} |", outcome(&r_h3), s4);
println!("(Honest inserts return Ok — nothing errors — and the reader never sees them.)");
// ── Table 3b: the liar refreshes as real time advances ──────────────
println!();
println!("TABLE 3b 2 s of real time pass; liar re-stamps now+(bound-1s); honest writes now");
tokio::time::sleep(std::time::Duration::from_secs(2)).await;
let t1 = now_us();
let r_l2 = n.arrive(&l, b"k", b"liar-refresh", t1 + shift - 1 * US).await; let s5 = who(&mut n, &l);
let r_h4 = n.arrive(&h, b"k", b"honest-4", t1).await; let s6 = who(&mut n, &l);
println!("| step | insert result | reader sees |");
println!("|---|---|---|");
println!("| liar re-stamps @t1+{} | {} | {} |", fmt_dur((shift - US) as i128), outcome(&r_l2), s5);
println!("| honest writes @t1 | {} | {} |", outcome(&r_h4), s6);
println!("(Each refresh is inside the bound at the moment it arrives, so the liar's hold on the key never has to expire.)");
// ── Control B: a liar who declares an honest timestamp loses ────────
{
let mut n = Net::new();
let (h, l) = (n.author(), n.author());
let t = now_us();
let _ = n.arrive(&l, b"k", b"liar-honest-stamp", t).await;
let _ = n.arrive(&h, b"k", b"honest-later", t + 1).await;
let w = n.winner(b"k").unwrap();
println!();
println!("CONTROL B liar with honest stamp, honest writes 1us later -> reader sees {} {}",
w.0, if w.0 == label_of(&h) { "(honest) OK" } else { "UNEXPECTED" });
}
// ── Control C: tie on timestamp is broken by content digest, not by arrival ─
{
let mut n1 = Net::new();
let (a, b) = (n1.author(), n1.author());
let t = now_us();
let _ = n1.arrive(&a, b"k", b"AAA", t).await;
let _ = n1.arrive(&b, b"k", b"BBB", t).await;
let w1 = n1.winner(b"k").unwrap();
// NOTE: single_latest_per_key across authors — check whether arrival order matters
let mut n2 = Net::new();
let ns2 = n2.ns.clone(); let _ = ns2;
let _ = n2.arrive(&b, b"k", b"BBB", t).await;
let _ = n2.arrive(&a, b"k", b"AAA", t).await;
let w2 = n2.winner(b"k").unwrap();
println!("CONTROL C equal timestamps, arrival A,B -> {}; arrival B,A -> {} {}",
w1.0, w2.0, if w1.0 == w2.0 { "(same, order-independent at equal stamp)" } else { "ORDER-DEPENDENT" });
}
Ok(())
}
fn label_of(a: &Author) -> String { format!("{}", a.id().fmt_short()) }
Cargo.toml and run.sh
[package]
name = "iroh-witness"
version = "0.1.0"
edition = "2021"
publish = false
# Pinned to the exact main commit that was read on 2026-09-09.
[dependencies]
iroh-docs = { git = "https://github.com/n0-computer/iroh-docs", rev = "8cfeacb087b4b195b1930683aa4448e990da0659", default-features = false }
iroh-blobs = { version = "0.103", default-features = false }
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }
rand = { version = "0.10", features = ["chacha"] }
anyhow = "1"
#!/usr/bin/env bash
# Builds and runs the witness inside the official Rust image. Nothing is
# installed on the host; the cargo registry and target dir are cached in
# named volumes so a second run is fast.
set -euo pipefail
cd "$(dirname "$0")"
docker run --rm \
-v "$PWD":/w -w /w \
-v iroh-witness-cargo:/usr/local/cargo/registry \
-v iroh-witness-git:/usr/local/cargo/git \
-v iroh-witness-target:/w/target \
rust:1-slim \
sh -c 'apt-get -qq update >/dev/null && apt-get -qq install -y pkg-config libssl-dev git >/dev/null 2>&1; cargo run --release --quiet' \
| tee RESULTS.raw.txt
Raw output of this run
iroh-docs main@8cfeacb MAX_TIMESTAMP_FUTURE_SHIFT = 600000000 us = 600 s
CONTROL A equal clocks: A@t accepted, B@t+1us accepted -> reader sees fda3c17136 (B, the later write) OK
TABLE 1 honest writer, clock ahead by Δ, receiver clock correct
| Δ | insert result |
|---|---|
| +1s | accepted |
| +1m0s | accepted |
| +9m59s | accepted |
| +10m0s | accepted |
| +10m1s | REJECTED TooFarInTheFuture |
| +1h | REJECTED TooFarInTheFuture |
| +24h | REJECTED TooFarInTheFuture |
TABLE 2 same writer: one write while clock was ahead by Δ, then clock corrected, writes again
| Δ | write while ahead | write after correction |
|---|---|---|
| +1m0s | accepted | REJECTED NewerEntryExists |
| +9m59s | accepted | REJECTED NewerEntryExists |
| +1h | REJECTED TooFarInTheFuture | accepted |
(NewerEntryExists means the writer's own corrected-clock writes are refused until real time passes its earlier stamp — a self-inflicted lockout of length Δ.)
TABLE 3 liar stamps now+(bound-1s); honest writers keep writing with correct clocks
| step | insert result | reader sees |
|---|---|---|
| honest writes @t0 | accepted | honest |
| liar writes @t0++9m59s | accepted | LIAR |
| honest writes @t0+1s | accepted | LIAR |
| honest writes @t0+1.5s | accepted | LIAR |
(Honest inserts return Ok — nothing errors — and the reader never sees them.)
TABLE 3b 2 s of real time pass; liar re-stamps now+(bound-1s); honest writes now
| step | insert result | reader sees |
|---|---|---|
| liar re-stamps @t1++9m59s | accepted | LIAR |
| honest writes @t1 | accepted | LIAR |
(Each refresh is inside the bound at the moment it arrives, so the liar's hold on the key never has to expire.)
CONTROL B liar with honest stamp, honest writes 1us later -> reader sees 2acde733d3 (honest) OK
CONTROL C equal timestamps, arrival A,B -> 4b2f7bdae6; arrival B,A -> 4b2f7bdae6 (same, order-independent at equal stamp)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with MAX_TIMESTAMP_FUTURE_SHIFT and the validation path in sync.rs, then inspect the gossip handling at engine/gossip.rs:181 and the witness in src/main.rs. Check the kv-crdts documentation page for its CRDT and timestamp descriptions. Done means the maintainer-approved behavior and the silent failure modes are explicitly documented, or the issue is clarified if a code change is intended.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems, documentation
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100