source-cooperative / source-cooperative/source.coop

Invalidate data-proxy cache when a model is updated

Open
#441 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
TypeScript
Stars
26
Forks
9
Avg merge
1d 7h
Merged PRs (30d)
42

Description

Problem

The data proxy (data.source.coop) caches connection/product metadata (currently up to ~5 minutes). When a user updates a model in the app — a product, a data connection, a product mirror's prefix, etc. — those changes don't take effect until the proxy's cache expires. This is confusing: the app reports the save succeeded, but reads through the proxy keep serving the old configuration.

As a stopgap the mirror-editing UI now shows a warning that changes may take up to 5 minutes to go live (see PR #440), but we should actually invalidate the cache on write rather than make users wait.

Proposal

On a successful mutation of a cached record (product, data connection, mirror, …), have source.coop tell the data proxy to drop that record from its cache so the next read repopulates it.

  • Add an endpoint on data.source.coop to clear the cache for a specific record (e.g. by connection id / product path).
  • Authorize the request using the auth token of the user making the request — the proxy should only clear a cache entry the caller is actually permitted to manage, reusing the same per-user authorization the proxy already applies to storage operations. This avoids giving source.coop a blanket admin key to flush arbitrary entries.
  • Call this endpoint from the relevant server actions after a successful write (product update, data-connection create/update/delete, add/remove/set-primary/prefix mirror actions).

Notes / open questions

  • Cache key granularity: is it per connection, per product, or both? The invalidation call needs to match whatever the proxy keys on.
  • Failure handling: if the invalidation call fails, the write has already succeeded — fall back to the existing TTL (and possibly surface the "may take up to 5 minutes" warning only in that case).
  • Cross-repo: the endpoint lives in the data-proxy repo; the calls live in source.coop. This issue tracks the source.coop side; a companion issue/PR may be needed on the proxy.

Filed from work on PR #440 (account-owned data connections + mirror prefix editing).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the relevant server actions for product updates, data-connection create/update/delete, and mirror changes, along with PR #440. Confirm the data proxy's cache-key and authorization requirements with the companion proxy work. Done means successful writes request per-record invalidation using the caller's auth token, while failed invalidation falls back to the existing TTL.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.