source-cooperative / source-cooperative/data.source.coop
[Proposed Feature] Send Cache-Control on proxied object responses
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 24
- Forks
- 6
- Avg merge
- 1h 32m
- Merged PRs (30d)
- 1
Description
Description of Feature:
Send a Cache-Control header on proxied object responses.
Responses today carry ETag and Last-Modified but no Cache-Control:
$ curl -sI https://data.source.coop/<repo>/collection.json
HTTP/2 200
content-type: application/json
etag: "6f56127110ec18360b527bcb3e0b707a"
last-modified: Tue, 15 Sep 2026 09:18:59 GMT
cf-cache-status: DYNAMIC
server: cloudflare
Without Cache-Control, RFC 9111 §4.2.2
permits a cache to pick its own freshness lifetime, and the common heuristic is
10% of the time since Last-Modified. A file untouched for ten days is then
treated as fresh for a day. Browsers apply this to fetch(), so a client that
re-reads metadata after a publish can get the previous version with no error and
no way to tell.
We hit this on a STAC catalog served from Source Cooperative. After publishing
updated item JSON, a browser kept returning the pre-publish bodies — different
ETags for the same URL — while curl returned current ones. The app dropped
26 of 27 archives from its picker, because each item looked like it advertised
no tileset. We now fetch the collection with a cache-busting query parameter
and key the item URLs off a version string in it, so a publish changes every
item URL. That works, but it is a workaround for a missing header, and the
collection itself is now uncacheable.
Revalidation already works, so the fix is cheap:
$ curl -sI -H 'If-None-Match: "6f56127110ec18360b527bcb3e0b707a"' \
https://data.source.coop/<repo>/collection.json
HTTP/2 304 # 0 bytes
S3 stores Cache-Control as object metadata and returns it on GET and HEAD, so
the proxy can pass through whatever the object carries. Objects we have checked
do not set it:
$ aws s3api head-object --bucket <bucket> --key <repo>/collection.json
{ "ContentType": "application/json", ... } # no CacheControl
Two options, not exclusive:
- Pass through
Cache-Controlfrom the object. Publishers then choose per
file —max-age=31536000, immutablefor data that is rewritten under a new
name,no-cachefor metadata that is overwritten in place. This costs the
proxy one header copy. - Send a default when the object sets none.
no-cacheis the safe one: it
permits storing but requires revalidation, and revalidation is already a
304 with an empty body.
Option 2 alone fixes the correctness problem. Option 1 lets publishers get
caching back for immutable data.
What value is this feature adding to Source Cooperative?
- Clients stop reading stale metadata after a publish. This is currently silent:
the response is a 200 with a plausible body, so nothing downstream detects it. - Publishers stop working around it with cache-busting query strings, which
force a full transfer every time and make caching impossible for anyone. - Whole-object reads of small metadata — STAC JSON, READMEs, styles, TileJSON —
become correctly cacheable by browsers and downstream caches, on the
publisher's terms instead of a heuristic's.
[!NOTE]
Scope, and how this relates to #188. This header alone does not make
ranged reads edge-cacheable, and an earlier version of this issue claimed it
did. Cloudflare's Cache API refuses to store a206
(docs), and
the proxy deliberately setsRequestCache::NoStoreon any subrequest carrying
aRangeheader (ForwardRequest::should_bypass_cachein multistore) so a
partial response can never poison the full-object cache entry. Making ranged
reads of large objects (COG, GeoParquet, PMTiles) fast is #188's job —
chunk-aligned caching inside the worker — and the two are complementary rather
than overlapping: #188 governs what the worker's own cache holds, this issue
governs what the proxy tells downstream caches.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading ForwardRequest::should_bypass_cache in multistore and trace how proxied object headers are built for GET and HEAD responses. Determine how object Cache-Control metadata and the no-header default should be handled, then verify that revalidation still returns 304 and that ranged requests remain outside this scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, rust
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100