n0-computer / n0-computer/iroh-docs

`MAX_TIMESTAMP_FUTURE_SHIFT` has no safe value: a writer 10 min 1 s ahead is silently refused everywhere, a writer at 9 min 59 s holds any key for as long as it keeps stamping

Open
#120 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
74
Forks
16
PR merge metrics
No merged PRs in 30d

Description

Summary. Conflicts are resolved by the writer's own timestamp
(Ord for Record, sync.rs:1158), gated only by a fixed 10-minute future bound
against the receiver's clock (sync.rs:640). I ran the two sides of that bound
through Replica::insert_remote_entry on main@8cfeacb. Both failure modes are
quiet, and they move in opposite directions with the same constant, so no value fixes
both.

Honest writer ahead. Accepted up to exactly +10 min 0 s; refused from +10 min 1 s
with TooFarInTheFuture. On the gossip path that refusal is
debug!("ignoring entry received via gossip") (engine/gossip.rs:181). The writer's
own replica accepted the same entry, so it sees a value its peers do not.

Honest writer whose clock gets corrected. One write while +1 min ahead, then
correct the clock: every subsequent write to that key is NewerEntryExists for one
minute. At +9 min 59 s, ten minutes. Fixing the clock is what locks you out.

Writer who lies inside the bound. Stamp now+9 min 59 s: accepted. Honest peers
keep writing with correct clocks: every insert returns Ok, and
single_latest_per_key never shows them. Two seconds later the liar stamps
now+9 min 59 s again: accepted. The hold never has to expire.

bound honest writer over it writer just under it
smaller refused sooner wins for less
larger refused later wins for longer

Controls. Equal clocks → later write wins. A liar with an honest stamp → loses to
a later honest write. Equal stamps → same winner whichever arrives first (the
content-digest tie-break is order-independent). So the stamp is the only input a peer
controls, and it is the whole decision.

What I am not claiming. Willow says "we are cognisant of their limitations, and
use them anyway,"
so the design is deliberate; I am not reporting a surprise. What I
could not find written down — in Willow, here, or the docs page that describes this as
built on CRDTs — is that the bound cannot be set safely, or that the honest-side
failure is silent. #97 fixed the unit slip that made the bound 600 ms; this is about
the 10 minutes it was meant to be. Not exercised: RBSR sessions and gossip transport
(both call insert_remote_entry); a receiver whose clock is slow (validate_entry
takes its own system_time_now(), so I could reason about it but not run it).

Reproduction. ~150 lines against the crate's public API, Docker, no
reimplementation: below. Every row above is in the
output.

If this is a known and accepted trade-off, a sentence saying so on the kv-crdts page

src/main.rs (the whole witness)
//! Witness: what iroh-docs does with a writer-declared timestamp.
//!
//! Everything below calls iroh-docs' own `Replica` API on its own in-memory
//! store, through the same `insert_remote_entry` path the gossip and sync
//! layers use. No reimplementation. Timestamps are chosen explicitly so the
//! writer's clock can be anything; the receiver's clock is the real one.
//!
//! Scope, stated narrowly: the replica insert/validate path and the
//! `single_latest_per_key` read. Not exercised: the network transport, the
//! range-based reconciliation session, or the blob layer.

use std::time::{SystemTime, UNIX_EPOCH};

use anyhow::Result;
use iroh_docs::{
    store::{Query, Store},
    sync::{ContentStatus, InsertError, Record, SignedEntry, ValidationFailure},
    Author, NamespaceSecret, MAX_TIMESTAMP_FUTURE_SHIFT,
};

const US: u64 = 1_000_000; // microseconds per second, the unit iroh-docs uses

fn now_us() -> u64 {
    SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_micros() as u64
}

fn fmt_dur(us: i128) -> String {
    let s = us / US as i128;
    if s.abs() >= 3600 { format!("{:+}h", s / 3600) }
    else if s.abs() >= 60 { format!("{:+}m{}s", s / 60, (s % 60).abs()) }
    else { format!("{:+}s", s) }
}

fn outcome(r: &std::result::Result<usize, InsertError>) -> &'static str {
    match r {
        Ok(_) => "accepted",
        Err(InsertError::Validation(ValidationFailure::TooFarInTheFuture)) => "REJECTED TooFarInTheFuture",
        Err(InsertError::NewerEntryExists) => "REJECTED NewerEntryExists",
        Err(_) => "REJECTED other",
    }
}

struct Net {
    store: Store,
    ns: NamespaceSecret,
    peer: [u8; 32],
}

impl Net {
    fn new() -> Self {
        let mut rng = rand::rng();
        Net { store: Store::memory(), ns: NamespaceSecret::new(&mut rng), peer: [7u8; 32] }
    }
    fn author(&self) -> Author { Author::new(&mut rand::rng()) }

    /// A remote writer's entry arriving at this receiver, with the timestamp
    /// the writer put on it.
    async fn arrive(&mut self, author: &Author, key: &[u8], data: &[u8], ts: u64)
        -> std::result::Result<usize, InsertError>
    {
        let rec = Record::new(iroh_blobs::Hash::new(data), data.len() as u64, ts);
        let e = SignedEntry::from_parts(&self.ns, author, key, rec);
        let mut replica = self.store.new_replica(self.ns.clone()).unwrap();
        replica.insert_remote_entry(e, self.peer, ContentStatus::Complete).await
    }

    /// What a reader of this replica sees as *the* value for `key`.
    fn winner(&mut self, key: &[u8]) -> Option<(String, u64)> {
        let q = Query::single_latest_per_key().key_exact(key);
        let mut it = self.store.get_many(self.ns.id(), q).unwrap();
        it.next().map(|e| { let e = e.unwrap(); (label(&e), e.timestamp()) })
    }
}

fn label(e: &SignedEntry) -> String {
    format!("{}", e.author().fmt_short())
}

#[tokio::main]
async fn main() -> Result<()> {
    let shift = MAX_TIMESTAMP_FUTURE_SHIFT;
    println!("iroh-docs main@8cfeacb  MAX_TIMESTAMP_FUTURE_SHIFT = {} us = {} s", shift, shift / US);
    println!();

    // ── Control A: honest writers, equal clocks ─────────────────────────
    {
        let mut n = Net::new();
        let (a, b) = (n.author(), n.author());
        let t = now_us();
        let ra = n.arrive(&a, b"k", b"A", t).await;
        let rb = n.arrive(&b, b"k", b"B", t + 1).await;
        let w = n.winner(b"k").unwrap();
        println!("CONTROL A  equal clocks: A@t {}, B@t+1us {} -> reader sees {}  {}",
            outcome(&ra), outcome(&rb), w.0,
            if w.0 == label_of(&b) { "(B, the later write) OK" } else { "UNEXPECTED" });
    }

    // ── Table 1: an honest writer whose clock is ahead by Δ ─────────────
    println!();
    println!("TABLE 1  honest writer, clock ahead by Δ, receiver clock correct");
    println!("| Δ | insert result |");
    println!("|---|---|");
    for &d in &[1 * US, 60 * US, shift - 1 * US, shift, shift + 1 * US, 3600 * US, 86400 * US] {
        let mut n = Net::new();
        let a = n.author();
        let r = n.arrive(&a, b"k", b"x", now_us() + d).await;
        println!("| {} | {} |", fmt_dur(d as i128), outcome(&r));
    }

    // ── Table 2: honest writer's clock was ahead, then got corrected ───
    println!();
    println!("TABLE 2  same writer: one write while clock was ahead by Δ, then clock corrected, writes again");
    println!("| Δ | write while ahead | write after correction |");
    println!("|---|---|---|");
    for &d in &[60 * US, shift - 1 * US, 3600 * US] {
        let mut n = Net::new();
        let a = n.author();
        let r1 = n.arrive(&a, b"k", b"while-ahead", now_us() + d).await;
        let r2 = n.arrive(&a, b"k", b"after-fix", now_us()).await;
        println!("| {} | {} | {} |", fmt_dur(d as i128), outcome(&r1), outcome(&r2));
    }
    println!("(NewerEntryExists means the writer's own corrected-clock writes are refused until real time passes its earlier stamp — a self-inflicted lockout of length Δ.)");

    // ── Table 3: a liar inside the bound ────────────────────────────────
    println!();
    println!("TABLE 3  liar stamps now+(bound-1s); honest writers keep writing with correct clocks");
    let mut n = Net::new();
    let (h, l) = (n.author(), n.author());
    let t0 = now_us();
    let who = |n: &mut Net, l: &Author| { let w = n.winner(b"k").unwrap().0; if w == label_of(l) { "LIAR" } else { "honest" } };
    let r_h1 = n.arrive(&h, b"k", b"honest-1", t0).await;            let s1 = who(&mut n, &l);
    let r_l  = n.arrive(&l, b"k", b"liar", t0 + shift - 1 * US).await; let s2 = who(&mut n, &l);
    let r_h2 = n.arrive(&h, b"k", b"honest-2", t0 + 1 * US).await;     let s3 = who(&mut n, &l);
    let r_h3 = n.arrive(&h, b"k", b"honest-3", t0 + 1_500_000).await;   let s4 = who(&mut n, &l);
    println!("| step | insert result | reader sees |");
    println!("|---|---|---|");
    println!("| honest writes @t0 | {} | {} |", outcome(&r_h1), s1);
    println!("| liar writes @t0+{} | {} | {} |", fmt_dur((shift - US) as i128), outcome(&r_l), s2);
    println!("| honest writes @t0+1s | {} | {} |", outcome(&r_h2), s3);
    println!("| honest writes @t0+1.5s | {} | {} |", outcome(&r_h3), s4);
    println!("(Honest inserts return Ok — nothing errors — and the reader never sees them.)");

    // ── Table 3b: the liar refreshes as real time advances ──────────────
    println!();
    println!("TABLE 3b  2 s of real time pass; liar re-stamps now+(bound-1s); honest writes now");
    tokio::time::sleep(std::time::Duration::from_secs(2)).await;
    let t1 = now_us();
    let r_l2 = n.arrive(&l, b"k", b"liar-refresh", t1 + shift - 1 * US).await; let s5 = who(&mut n, &l);
    let r_h4 = n.arrive(&h, b"k", b"honest-4", t1).await;                       let s6 = who(&mut n, &l);
    println!("| step | insert result | reader sees |");
    println!("|---|---|---|");
    println!("| liar re-stamps @t1+{} | {} | {} |", fmt_dur((shift - US) as i128), outcome(&r_l2), s5);
    println!("| honest writes @t1 | {} | {} |", outcome(&r_h4), s6);
    println!("(Each refresh is inside the bound at the moment it arrives, so the liar's hold on the key never has to expire.)");

    // ── Control B: a liar who declares an honest timestamp loses ────────
    {
        let mut n = Net::new();
        let (h, l) = (n.author(), n.author());
        let t = now_us();
        let _ = n.arrive(&l, b"k", b"liar-honest-stamp", t).await;
        let _ = n.arrive(&h, b"k", b"honest-later", t + 1).await;
        let w = n.winner(b"k").unwrap();
        println!();
        println!("CONTROL B  liar with honest stamp, honest writes 1us later -> reader sees {} {}",
            w.0, if w.0 == label_of(&h) { "(honest) OK" } else { "UNEXPECTED" });
    }

    // ── Control C: tie on timestamp is broken by content digest, not by arrival ─
    {
        let mut n1 = Net::new();
        let (a, b) = (n1.author(), n1.author());
        let t = now_us();
        let _ = n1.arrive(&a, b"k", b"AAA", t).await;
        let _ = n1.arrive(&b, b"k", b"BBB", t).await;
        let w1 = n1.winner(b"k").unwrap();
        // NOTE: single_latest_per_key across authors — check whether arrival order matters
        let mut n2 = Net::new();
        let ns2 = n2.ns.clone(); let _ = ns2;
        let _ = n2.arrive(&b, b"k", b"BBB", t).await;
        let _ = n2.arrive(&a, b"k", b"AAA", t).await;
        let w2 = n2.winner(b"k").unwrap();
        println!("CONTROL C  equal timestamps, arrival A,B -> {}; arrival B,A -> {}  {}",
            w1.0, w2.0, if w1.0 == w2.0 { "(same, order-independent at equal stamp)" } else { "ORDER-DEPENDENT" });
    }

    Ok(())
}

fn label_of(a: &Author) -> String { format!("{}", a.id().fmt_short()) }
Cargo.toml and run.sh
[package]
name = "iroh-witness"
version = "0.1.0"
edition = "2021"
publish = false

# Pinned to the exact main commit that was read on 2026-09-09.
[dependencies]
iroh-docs = { git = "https://github.com/n0-computer/iroh-docs", rev = "8cfeacb087b4b195b1930683aa4448e990da0659", default-features = false }
iroh-blobs = { version = "0.103", default-features = false }
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }
rand = { version = "0.10", features = ["chacha"] }
anyhow = "1"
#!/usr/bin/env bash
# Builds and runs the witness inside the official Rust image. Nothing is
# installed on the host; the cargo registry and target dir are cached in
# named volumes so a second run is fast.
set -euo pipefail
cd "$(dirname "$0")"
docker run --rm \
  -v "$PWD":/w -w /w \
  -v iroh-witness-cargo:/usr/local/cargo/registry \
  -v iroh-witness-git:/usr/local/cargo/git \
  -v iroh-witness-target:/w/target \
  rust:1-slim \
  sh -c 'apt-get -qq update >/dev/null && apt-get -qq install -y pkg-config libssl-dev git >/dev/null 2>&1; cargo run --release --quiet' \
  | tee RESULTS.raw.txt
Raw output of this run
iroh-docs main@8cfeacb  MAX_TIMESTAMP_FUTURE_SHIFT = 600000000 us = 600 s

CONTROL A  equal clocks: A@t accepted, B@t+1us accepted -> reader sees fda3c17136  (B, the later write) OK

TABLE 1  honest writer, clock ahead by Δ, receiver clock correct
| Δ | insert result |
|---|---|
| +1s | accepted |
| +1m0s | accepted |
| +9m59s | accepted |
| +10m0s | accepted |
| +10m1s | REJECTED TooFarInTheFuture |
| +1h | REJECTED TooFarInTheFuture |
| +24h | REJECTED TooFarInTheFuture |

TABLE 2  same writer: one write while clock was ahead by Δ, then clock corrected, writes again
| Δ | write while ahead | write after correction |
|---|---|---|
| +1m0s | accepted | REJECTED NewerEntryExists |
| +9m59s | accepted | REJECTED NewerEntryExists |
| +1h | REJECTED TooFarInTheFuture | accepted |
(NewerEntryExists means the writer's own corrected-clock writes are refused until real time passes its earlier stamp — a self-inflicted lockout of length Δ.)

TABLE 3  liar stamps now+(bound-1s); honest writers keep writing with correct clocks
| step | insert result | reader sees |
|---|---|---|
| honest writes @t0 | accepted | honest |
| liar writes @t0++9m59s | accepted | LIAR |
| honest writes @t0+1s | accepted | LIAR |
| honest writes @t0+1.5s | accepted | LIAR |
(Honest inserts return Ok — nothing errors — and the reader never sees them.)

TABLE 3b  2 s of real time pass; liar re-stamps now+(bound-1s); honest writes now
| step | insert result | reader sees |
|---|---|---|
| liar re-stamps @t1++9m59s | accepted | LIAR |
| honest writes @t1 | accepted | LIAR |
(Each refresh is inside the bound at the moment it arrives, so the liar's hold on the key never has to expire.)

CONTROL B  liar with honest stamp, honest writes 1us later -> reader sees 2acde733d3 (honest) OK
CONTROL C  equal timestamps, arrival A,B -> 4b2f7bdae6; arrival B,A -> 4b2f7bdae6  (same, order-independent at equal stamp)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with MAX_TIMESTAMP_FUTURE_SHIFT and the validation path in sync.rs, then inspect the gossip handling at engine/gossip.rs:181 and the witness in src/main.rs. Check the kv-crdts documentation page for its CRDT and timestamp descriptions. Done means the maintainer-approved behavior and the silent failure modes are explicitly documented, or the issue is clarified if a code change is intended.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems, documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.