Extract connector and explorer logic into a standalone shared library
- Dominant language
- TypeScript
- Stars
- 481
- Forks
- 108
- Avg merge
- 6d 8h
- Merged PRs (30d)
- 5
Description
## Summary
Extract the database connector and explorer logic from `packages/graph-explorer/src/connector/` into a new standalone package within the monorepo (e.g., `packages/graph-explorer-client`). This library would provide a pure TypeScript API for querying graph databases (Gremlin, openCypher, SPARQL) with no dependency on React, Jotai, TanStack Query, or any other UI framework. It must run in both browser and Node.js environments.
## Motivation
The connector layer currently lives inside the React frontend package and is entangled with UI-specific concerns (TanStack Query cache management, Jotai atoms, React hooks). Extracting it would:
- Enable server-side usage (e.g., the proxy server could query databases directly, or a CLI tool could be built)
- Allow other frontend frameworks or applications to use the same query logic
- Enforce a clean separation between "how to talk to graph databases" and "how to render graph data"
- Make the query/mapping logic independently testable without any UI dependencies
- Simplify the `graph-explorer` package by reducing its scope to UI concerns
## Current Architecture Analysis
The connector layer has two distinct sub-layers today:
### 1. Pure Query Logic (environment-agnostic)
These have no inherent dependency on React or browser APIs beyond `fetch`:
- **Explorer implementations**: `gremlinExplorer.ts`, `sparqlExplorer.ts`, `openCypherExplorer.ts`
- **Query templates**: `gremlin/fetchNeighbors/oneHopTemplate.ts`, `sparql/keywordSearch/keywordSearchTemplate.ts`, etc.
- **Response mappers**: `gremlin/mappers/`, `sparql/mappers/`, `openCypher/mappers/`
- **The `Explorer` interface** in `useGEFetchTypes.ts` — already a clean abstraction that defines the API surface
- **Core entity types**: `Vertex`, `Edge`, `Entities`, branded ID types (`VertexId`, `EdgeId`, `VertexType`, `EdgeType`)
- **Transport**: `fetchDatabaseRequest.ts` — HTTP transport with Neptune error parsing, auth headers, proxy logic
### 2. React Integration Layer (should stay in `graph-explorer`)
These are tightly coupled to React/Jotai/TanStack Query:
- **`queries/` folder**: TanStack Query `queryOptions` wrappers (`searchQuery.ts`, `schemaSyncQuery.ts`, etc.)
- **`queries/helpers.ts`**: References `explorerAtom`, `nodesAtom`, `edgesAtom`, Jotai store, QueryClient cache
- **`@/core/connector.ts`**: The `explorerAtom` and `useExplorer()` hook that wires the explorer to React state
- **`connector/entities/bundle.ts`**: Has a display formatting function that depends on `@/hooks` for `TextTransformer`
## Proposed Separation of Concerns
### New Package: `packages/graph-explorer-client`
**Owns:**
- The `Explorer` interface (renamed from `useGEFetchTypes.ts` to something like `explorer.ts`)
- All request/response types (`NeighborsRequest`, `SchemaResponse`, `KeywordSearchRequest`, etc.)
- Core entity types (`Vertex`, `Edge`, `Entities`, `VertexId`, `EdgeId`, `VertexType`, `EdgeType`, `EntityProperties`, etc.)
- Branded type utility
- Explorer factory functions (`createGremlinExplorer`, `createSparqlExplorer`, `createOpenCypherExplorer`)
- All query template builders (Gremlin traversals, SPARQL queries, openCypher queries)
- All response mappers (Gremlin JSON → Vertex/Edge, SPARQL bindings → Vertex/Edge, etc.)
- Query language-specific types (`GVertex`, `GEdge`, `SparqlValue`, `OCVertex`, etc.)
- Shared utilities needed by the above: `query` template tag, `escapeString`, `batchPromisesSerially`, `NetworkError`, constants (`DEFAULT_BATCH_REQUEST_SIZE`, `DEFAULT_SAMPLE_SIZE`, etc.)
- `ConnectionConfig` and `NormalizedConnection` types (currently in `@shared/types` and `@/core`)
- `LoggerConnector` interface (but not the `ServerLoggerConnector` implementation that uses `fetch` to the proxy)
**Does NOT own:**
- TanStack Query wrappers (`queries/` folder)
- Jotai atoms or React hooks
- Display formatting logic (e.g., `getDisplayValueForBundle` with `TextTransformer`)
- Proxy server authentication/header logic (this could be injected)
### Stays in `graph-explorer`
- `connector/queries/` — TanStack Query integration layer
- `core/connector.ts` — `explorerAtom`, `useExplorer()`, `useQueryEngine()`
- `connector/entities/bundle.ts` display functions (or the display part moves to a UI utility)
- All React components, hooks, and state management
## Key Design Decisions to Make
### 1. Transport Abstraction
`fetchDatabaseRequest` currently handles Neptune-specific concerns (auth headers, proxy headers, error parsing). The new library needs a transport abstraction:
```typescript
// Option A: Accept a generic fetch function
type DatabaseFetch = (query: string) => Promise;
// Option B: Accept a transport interface
interface Transport {
execute(query: string, options?: RequestOptions): Promise;
}
```
The current explorers already use this pattern internally (e.g., `GremlinFetch`, `SparqlFetch`, `OpenCypherFetch` types). This could be formalized as the injection point.
### 2. Logger Injection
The explorers currently import `logger` from `@/utils` and accept a `LoggerConnector` for remote logging. The new library should define a `Logger` interface and accept it via dependency injection rather than importing a global.
### 3. SPARQL BlankNodesMap
`createSparqlExplorer` currently accepts a `BlankNodesMap` parameter for stateful blank node tracking. This is a design smell — the new library should either:
- Manage blank node state internally within the explorer instance
- Define a clear interface for blank node storage that consumers provide
### 4. Result Entity Types
The `connector/entities/` types (`ResultVertex`, `ResultEdge`, `ResultScalar`, `ResultBundle`) represent query results before they are fully materialized. These are part of the query abstraction and should move to the new library. However, the display formatting functions (`getDisplayValueForBundle`, `getDisplayValueForScalar`) that depend on `TextTransformer` should stay in the UI layer.
### 5. FeatureFlags
The explorers currently receive `FeatureFlags` (containing `showDebugActions` and `allowLoggingDbQuery`). The library should accept only the flags it actually needs (e.g., `allowLoggingDbQuery` for the proxy header), not the full UI feature flags type.
## Suggested Migration Strategy
1. Create the new `packages/graph-explorer-client` package with its own `package.json`, `tsconfig.json`, and test config
2. Move core entity types first (`Vertex`, `Edge`, `Entities`, branded IDs, `EntityProperties`) — these have the fewest dependencies
3. Move the `Explorer` interface and all request/response types
4. Move shared utilities (`query`, `escapeString`, `batchPromisesSerially`, `NetworkError`, constants)
5. Move query templates and response mappers for each language
6. Move explorer factory functions, introducing the transport abstraction
7. Update `graph-explorer` to import from the new package
8. Update `@shared/types` — `ConnectionConfig` and `QueryEngine` should move to the new library since they define the connection contract
## Open Questions
- Package name: `graph-explorer-client`, `graph-query-client`, or something else?
- Should the `ConnectionConfig` type live in this new library or remain in `@shared/types`?
- Should the library include a default `fetch`-based transport, or should transport always be injected?
- How should the library handle the `uuid` dependency currently used for `queryId` generation in the explorers?
Contributor guide
Research direction
Start by reading packages/graph-explorer/src/connector/ and the Explorer interface in useGEFetchTypes.ts, then trace the query templates, mappers, transport, and React integration boundaries listed in the issue. Done means a new packages/graph-explorer-client package owns the framework-independent API and graph query logic, while graph-explorer retains TanStack Query, Jotai, hooks, and display formatting without UI dependencies in the client.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- api, backend-api-design
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100