AWS: integration's fixed 10s context deadline stalls the append path on large logs, and is not configurable

Summary

integrateEntriesJob gives each integration cycle a fixed 10s context that cannot be overridden. Inside that budget, Prewarm issues a number of S3 GetObject calls, and there is no per-request timeout or retry — so a single slow object discards the work of the entire cycle, and the failure surfaces as a context deadline exceeded on whichever GET happened to be in flight.

We hit this in production on a log of ~13M entries. I want to be precise about what I am and am not claiming: I have not established that this caused our outage. It is a sharp edge we ran into, with what looks to me like a design smell, and the reporting below separates the two.

Where

storage/aws/aws.go, integrateEntriesJob:

ctx, cancel := context.WithTimeout(ctx, defaultIntegrationTimeout)
defer cancel()
if _, err := a.sequencer.consumeEntries(ctx, DefaultIntegrationSizeLimit, a.integrateEntries, false); err != nil {
    return err
}

with

// defaultAssignEntriesTimeout is the default context timeout applied when assigning a batch of entries to the MySQL sequencer.
defaultAssignEntriesTimeout = 2 * time.Second
// defaultIntegrationTimeout is the default context timeout applied when undertaking an integration task.
defaultIntegrationTimeout = 10 * time.Second
// defaultGCTimeout is the default context timeout applied when undertaking a garbage collection task.
defaultGCTimeout = 30 * time.Second

These are named default*, but nothing can override them — there is no AppendOptions setter the way there is for e.g. GarbageCollectionInterval. In v1.0.2 (which rekor-tiles v2.3.0 ships) it is an inline context.WithTimeout(ctx, 10*time.Second) literal.

What we observed

level=ERROR msg="integrateEntries: integrate: storage.Integrate: failed to create range covering existing log:
Prewarm: getTiles: getObject: failed to create reader for object \"tile/2/000.p/202\" in bucket \"<bucket>\":
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded"
level=ERROR msg="failed to rollback Tx: sql: transaction has already been committed or rolled back"

Checked before reporting:

• The object exists — tile/2/000.p/202, 6464 bytes, HeadObject returns 200. Not a missing tile.
• StatusCode: 0 with an empty RequestID means no HTTP response was ever received. The GET was abandoned at the deadline, not rejected by S3.
• S3 was otherwise healthy — no SlowDown, no 5xx, and the same bucket served normal traffic throughout.
• These errors are rare, not continuous. integrateEntriesJob ticks every second; over a 45-minute window we logged three of them. So the 10s budget is not being blown routinely — it is a cliff that gets hit when an individual S3 read is transiently slow.

The actual concern

Not that 10s is too small on average. Our normal end-to-end add latency is ~2.1s and has been flat for over a week.

The concern is the shape:

1. One deadline covers an unbounded number of network round-trips. Prewarm fetches the tiles covering the existing tree; each is an S3 GET. There is no per-request timeout and no retry inside getTiles, so one unlucky object throws away every other fetch in that cycle.
2. It is not tunable. An operator who sees this has no lever at all short of patching and rebuilding. That is what pushed us to open this issue rather than adjust a flag.
3. The failure is silent from the outside. Integration stalling does not affect the gRPC health service, so an orchestrator sees a perfectly healthy task while Add futures never resolve and clients time out. (Happy to file that separately if it is a distinct concern.)

What would help, in rough order of preference

1. Make these settable via AppendOptions, as GarbageCollectionInterval already is.
2. Per-GetObject timeout plus a bounded retry inside getTiles, rather than one deadline spanning the whole cycle. A transient slow read should cost a retry, not an integration round.
3. Optionally, derive the budget from the number of tiles about to be fetched.

Also worth a look while you are in there: failed to rollback Tx: sql: transaction has already been committed or rolled back is logged alongside every one of these failures, which suggests the error path in consumeEntries double-handles the transaction.

Environment

• tessera v1.0.2, via rekor-tiles v2.3.0, AWS driver (S3 + Aurora MySQL), single region
• Same 10s value on main (a820582) as defaultIntegrationTimeout
• Log ~13M entries; ~2.1s mean add latency, p99 ~4-5s

Happy to send a PR for (1) and/or (2) if that shape is agreeable.