aws / aws/graph-explorer

Spike: Decouple schema sync from TanStack Query data cache

Open
#1,616 0 comments 0 reactions 1 assignee Claimed by @kmcginnes View on GitHub
Dominant language
TypeScript
Stars
481
Forks
108
Avg merge
6d 8h
Merged PRs (30d)
5

Description

## Goal

Resolve the dual-cache contention between TanStack Query and Jotai/localforage for schema data. Currently, schema sync uses TQ as both an orchestrator and a data cache, while Jotai/localforage is also the persistent cache for the same data. This creates circular dependencies and complex synchronization logic that is difficult to maintain.

The spike should determine whether the proposed approach (below) is viable and identify any edge cases or blockers.

## Background

The schema sync process has grown into a complex system where two caches fight over the same data:

1. **Jotai + localforage** (`schemaAtom` via `atomWithLocalForage`) — the persistent, authoritative schema store
2. **TanStack Query cache** — the transient query cache that also holds schema data

The contention manifests in several ways:

- `schemaSyncQuery` uses `initialData: activeSchema` to seed TQ from Jotai, then writes back to Jotai via `store.set(replaceSchemaAtom, ...)`, then reads it back with `store.get(activeSchemaAtom)` to return to TQ
- `edgeConnectionsQuery` writes to Jotai via `store.set(setEdgeConnectionsAtom, ...)` then manually syncs back to TQ via `client.setQueryData(schemaSyncQuery(...).queryKey, newSchema)`
- `useUpdateSchemaFromEntities` writes directly to Jotai but TQ does not know about it, so the TQ cache goes stale
- `useMaybeActiveSchema()` reads from Jotai (via `useDeferredValue`), which is passed into `schemaSyncQuery()` as `initialData`, creating a circular dependency

## Proposed Approach

Keep TanStack Query for orchestration (retry, abort, status tracking, deduplication) but stop using it as a data store for schema. Three changes:

### 1. Drop `initialData` from `schemaSyncQuery`

Stop seeding TQ from Jotai. TQ should not hold schema data at all.

```ts
export function schemaSyncQuery(hasConnection: boolean, hasSyncFail: boolean) {
return queryOptions({
queryKey: ["schema", "sync"],
staleTime: Infinity,
enabled: hasConnection && !hasSyncFail,
retry: 3,
queryFn: async ({ signal, meta }) => {
const store = getStore(meta);
const explorer = store.get(explorerAtom);

const schema = await explorer.fetchSchema({ signal });
const prefixes = generateSchemaPrefixes(getSchemaUris(schema), []);
store.set(replaceSchemaAtom, schema, prefixes);
// Return void — TQ holds status only, not schema data
},
});
}
```

### 2. Drop `client.setQueryData` cross-sync from `edgeConnectionsQuery`

Edge connections write to Jotai only. No manual sync back to the schema query cache.

```ts
export function edgeConnectionsQuery(hasSchema: boolean, hasSyncFail: boolean, edgeTypes: EdgeType[]) {
return queryOptions({
queryKey: ["schema", "edgeConnections", edgeTypes],
staleTime: Infinity,
enabled: hasSchema && !hasSyncFail,
retry: 3,
queryFn: async ({ signal, meta }) => {
const store = getStore(meta);
const explorer = store.get(explorerAtom);

const result = await explorer.fetchEdgeConnections({ edgeTypes }, { signal });
store.set(setEdgeConnectionsAtom, result.edgeConnections);
// Return void — TQ holds status only
},
});
}
```

### 3. Update consumers to read schema from Jotai, status from TQ

- `SchemaDiscoveryBoundary` reads `isFetching`/`error` from TQ queries, reads schema existence from `useHasActiveSchema()` (Jotai)
- `ConnectionDetail` reads sync status from TQ, schema data from Jotai
- `SchemaGraphToolbar` reads `isFetching` from TQ for the refresh button state
- `useSchemaSync` simplifies — passes scalar booleans to query options instead of the full schema object

### 4. Simplify `useSchemaSync` hook

The hook no longer needs to pass the full `activeSchema` object into query options. It passes derived booleans:

```ts
export function useSchemaSync() {
const config = useConfiguration();
const schema = useMaybeActiveSchema();

const schemaDiscoveryQuery = useQuery(
schemaSyncQuery(config != null, schema?.lastSyncFail ?? false),
);
const edgeDiscoveryQuery = useQuery(
edgeConnectionsQuery(
schema?.lastUpdate != null,
schema?.lastEdgeConnectionSyncFail ?? false,
schema?.edges.map(e => e.type).toSorted() ?? [],
),
);

// ...
}
```

## Files Affected

### Core changes
- `src/connector/queries/schemaSyncQuery.ts` — remove `initialData`, return `void`
- `src/connector/queries/edgeConnectionsQuery.ts` — remove `initialData`, remove `client.setQueryData`, return `void`
- `src/hooks/useSchemaSync.ts` — pass scalar booleans instead of full schema object

### Consumer updates
- `src/components/SchemaDiscoveryBoundary.tsx` — read schema from Jotai, status from TQ
- `src/modules/ConnectionDetail/ConnectionDetail.tsx` — minor adjustments to status reads
- `src/modules/SchemaGraph/SchemaGraphToolbar.tsx` — no change expected (`isFetching` stays)

### Test updates
- `src/connector/queries/schemaSyncQuery.test.ts` — update for void return, no `initialData`
- `src/connector/queries/edgeConnectionsQuery.test.ts` — update for void return, no `setQueryData`
- `src/hooks/useSchemaSync.test.ts` — update for new hook signature
- `src/components/SchemaDiscoveryBoundary.test.tsx` — update for new data flow

## Constraints

- `SchemaStorageModel` shape must not change — existing users have persisted schemas in IndexedDB that must load without migration
- `useUpdateSchemaFromEntities` (incremental schema growth) should not change — it already writes directly to Jotai, which is the correct pattern
- Other TQ queries (search, vertex details, neighbor counts, etc.) are unaffected — they are genuine cacheable request/response pairs

## Expected Outcome

- A proof of concept branch demonstrating the approach works end-to-end
- Confirmation that existing persisted schemas load correctly without re-sync
- Identification of any edge cases with the TQ status-only pattern (e.g., stale `isFetching` after navigation)
- Recommendation on whether to proceed with a full implementation

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.