source-cooperative / source-cooperative/data.source.coop

[Proposed Feature] Send Cache-Control on proxied object responses

Open
#225 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
24
Forks
6
Avg merge
1h 32m
Merged PRs (30d)
1

Description

Description of Feature:

Send a Cache-Control header on proxied object responses.

Responses today carry ETag and Last-Modified but no Cache-Control:

$ curl -sI https://data.source.coop/<repo>/collection.json
HTTP/2 200
content-type: application/json
etag: "6f56127110ec18360b527bcb3e0b707a"
last-modified: Tue, 15 Sep 2026 09:18:59 GMT
cf-cache-status: DYNAMIC
server: cloudflare

Without Cache-Control, RFC 9111 §4.2.2
permits a cache to pick its own freshness lifetime, and the common heuristic is
10% of the time since Last-Modified. A file untouched for ten days is then
treated as fresh for a day. Browsers apply this to fetch(), so a client that
re-reads metadata after a publish can get the previous version with no error and
no way to tell.

We hit this on a STAC catalog served from Source Cooperative. After publishing
updated item JSON, a browser kept returning the pre-publish bodies — different
ETags for the same URL — while curl returned current ones. The app dropped
26 of 27 archives from its picker, because each item looked like it advertised
no tileset. We now fetch the collection with a cache-busting query parameter
and key the item URLs off a version string in it, so a publish changes every
item URL. That works, but it is a workaround for a missing header, and the
collection itself is now uncacheable.

Revalidation already works, so the fix is cheap:

$ curl -sI -H 'If-None-Match: "6f56127110ec18360b527bcb3e0b707a"' \
    https://data.source.coop/<repo>/collection.json
HTTP/2 304      # 0 bytes

S3 stores Cache-Control as object metadata and returns it on GET and HEAD, so
the proxy can pass through whatever the object carries. Objects we have checked
do not set it:

$ aws s3api head-object --bucket <bucket> --key <repo>/collection.json
{ "ContentType": "application/json", ... }   # no CacheControl

Two options, not exclusive:

  1. Pass through Cache-Control from the object. Publishers then choose per
    file — max-age=31536000, immutable for data that is rewritten under a new
    name, no-cache for metadata that is overwritten in place. This costs the
    proxy one header copy.
  2. Send a default when the object sets none. no-cache is the safe one: it
    permits storing but requires revalidation, and revalidation is already a
    304 with an empty body.

Option 2 alone fixes the correctness problem. Option 1 lets publishers get
caching back for immutable data.

What value is this feature adding to Source Cooperative?

  • Clients stop reading stale metadata after a publish. This is currently silent:
    the response is a 200 with a plausible body, so nothing downstream detects it.
  • Publishers stop working around it with cache-busting query strings, which
    force a full transfer every time and make caching impossible for anyone.
  • Whole-object reads of small metadata — STAC JSON, READMEs, styles, TileJSON —
    become correctly cacheable by browsers and downstream caches, on the
    publisher's terms instead of a heuristic's.

[!NOTE]
Scope, and how this relates to #188. This header alone does not make
ranged reads edge-cacheable, and an earlier version of this issue claimed it
did. Cloudflare's Cache API refuses to store a 206
(docs), and
the proxy deliberately sets RequestCache::NoStore on any subrequest carrying
a Range header (ForwardRequest::should_bypass_cache in multistore) so a
partial response can never poison the full-object cache entry. Making ranged
reads of large objects (COG, GeoParquet, PMTiles) fast is #188's job —
chunk-aligned caching inside the worker — and the two are complementary rather
than overlapping: #188 governs what the worker's own cache holds, this issue
governs what the proxy tells downstream caches.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading ForwardRequest::should_bypass_cache in multistore and trace how proxied object headers are built for GET and HEAD responses. Determine how object Cache-Control metadata and the no-header default should be handled, then verify that revalidation still returns 304 and that ranged requests remain outside this scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.