aws / aws/graph-explorer

Extract connector and explorer logic into a standalone shared library

Open
#1,558 0 comments 0 reactions 0 assignees View on GitHub
enhancement internal needs-triage tech debt
Dominant language
TypeScript
Stars
481
Forks
108
Avg merge
6d 8h
Merged PRs (30d)
5

Description

## Summary

Extract the database connector and explorer logic from `packages/graph-explorer/src/connector/` into a new standalone package within the monorepo (e.g., `packages/graph-explorer-client`). This library would provide a pure TypeScript API for querying graph databases (Gremlin, openCypher, SPARQL) with no dependency on React, Jotai, TanStack Query, or any other UI framework. It must run in both browser and Node.js environments.

## Motivation

The connector layer currently lives inside the React frontend package and is entangled with UI-specific concerns (TanStack Query cache management, Jotai atoms, React hooks). Extracting it would:

- Enable server-side usage (e.g., the proxy server could query databases directly, or a CLI tool could be built)
- Allow other frontend frameworks or applications to use the same query logic
- Enforce a clean separation between "how to talk to graph databases" and "how to render graph data"
- Make the query/mapping logic independently testable without any UI dependencies
- Simplify the `graph-explorer` package by reducing its scope to UI concerns

## Current Architecture Analysis

The connector layer has two distinct sub-layers today:

### 1. Pure Query Logic (environment-agnostic)

These have no inherent dependency on React or browser APIs beyond `fetch`:

- **Explorer implementations**: `gremlinExplorer.ts`, `sparqlExplorer.ts`, `openCypherExplorer.ts`
- **Query templates**: `gremlin/fetchNeighbors/oneHopTemplate.ts`, `sparql/keywordSearch/keywordSearchTemplate.ts`, etc.
- **Response mappers**: `gremlin/mappers/`, `sparql/mappers/`, `openCypher/mappers/`
- **The `Explorer` interface** in `useGEFetchTypes.ts` — already a clean abstraction that defines the API surface
- **Core entity types**: `Vertex`, `Edge`, `Entities`, branded ID types (`VertexId`, `EdgeId`, `VertexType`, `EdgeType`)
- **Transport**: `fetchDatabaseRequest.ts` — HTTP transport with Neptune error parsing, auth headers, proxy logic

### 2. React Integration Layer (should stay in `graph-explorer`)

These are tightly coupled to React/Jotai/TanStack Query:

- **`queries/` folder**: TanStack Query `queryOptions` wrappers (`searchQuery.ts`, `schemaSyncQuery.ts`, etc.)
- **`queries/helpers.ts`**: References `explorerAtom`, `nodesAtom`, `edgesAtom`, Jotai store, QueryClient cache
- **`@/core/connector.ts`**: The `explorerAtom` and `useExplorer()` hook that wires the explorer to React state
- **`connector/entities/bundle.ts`**: Has a display formatting function that depends on `@/hooks` for `TextTransformer`

## Proposed Separation of Concerns

### New Package: `packages/graph-explorer-client`

**Owns:**
- The `Explorer` interface (renamed from `useGEFetchTypes.ts` to something like `explorer.ts`)
- All request/response types (`NeighborsRequest`, `SchemaResponse`, `KeywordSearchRequest`, etc.)
- Core entity types (`Vertex`, `Edge`, `Entities`, `VertexId`, `EdgeId`, `VertexType`, `EdgeType`, `EntityProperties`, etc.)
- Branded type utility
- Explorer factory functions (`createGremlinExplorer`, `createSparqlExplorer`, `createOpenCypherExplorer`)
- All query template builders (Gremlin traversals, SPARQL queries, openCypher queries)
- All response mappers (Gremlin JSON → Vertex/Edge, SPARQL bindings → Vertex/Edge, etc.)
- Query language-specific types (`GVertex`, `GEdge`, `SparqlValue`, `OCVertex`, etc.)
- Shared utilities needed by the above: `query` template tag, `escapeString`, `batchPromisesSerially`, `NetworkError`, constants (`DEFAULT_BATCH_REQUEST_SIZE`, `DEFAULT_SAMPLE_SIZE`, etc.)
- `ConnectionConfig` and `NormalizedConnection` types (currently in `@shared/types` and `@/core`)
- `LoggerConnector` interface (but not the `ServerLoggerConnector` implementation that uses `fetch` to the proxy)

**Does NOT own:**
- TanStack Query wrappers (`queries/` folder)
- Jotai atoms or React hooks
- Display formatting logic (e.g., `getDisplayValueForBundle` with `TextTransformer`)
- Proxy server authentication/header logic (this could be injected)

### Stays in `graph-explorer`

- `connector/queries/` — TanStack Query integration layer
- `core/connector.ts` — `explorerAtom`, `useExplorer()`, `useQueryEngine()`
- `connector/entities/bundle.ts` display functions (or the display part moves to a UI utility)
- All React components, hooks, and state management

## Key Design Decisions to Make

### 1. Transport Abstraction

`fetchDatabaseRequest` currently handles Neptune-specific concerns (auth headers, proxy headers, error parsing). The new library needs a transport abstraction:

```typescript
// Option A: Accept a generic fetch function
type DatabaseFetch = (query: string) => Promise;

// Option B: Accept a transport interface
interface Transport {
execute(query: string, options?: RequestOptions): Promise;
}
```

The current explorers already use this pattern internally (e.g., `GremlinFetch`, `SparqlFetch`, `OpenCypherFetch` types). This could be formalized as the injection point.

### 2. Logger Injection

The explorers currently import `logger` from `@/utils` and accept a `LoggerConnector` for remote logging. The new library should define a `Logger` interface and accept it via dependency injection rather than importing a global.

### 3. SPARQL BlankNodesMap

`createSparqlExplorer` currently accepts a `BlankNodesMap` parameter for stateful blank node tracking. This is a design smell — the new library should either:
- Manage blank node state internally within the explorer instance
- Define a clear interface for blank node storage that consumers provide

### 4. Result Entity Types

The `connector/entities/` types (`ResultVertex`, `ResultEdge`, `ResultScalar`, `ResultBundle`) represent query results before they are fully materialized. These are part of the query abstraction and should move to the new library. However, the display formatting functions (`getDisplayValueForBundle`, `getDisplayValueForScalar`) that depend on `TextTransformer` should stay in the UI layer.

### 5. FeatureFlags

The explorers currently receive `FeatureFlags` (containing `showDebugActions` and `allowLoggingDbQuery`). The library should accept only the flags it actually needs (e.g., `allowLoggingDbQuery` for the proxy header), not the full UI feature flags type.

## Suggested Migration Strategy

1. Create the new `packages/graph-explorer-client` package with its own `package.json`, `tsconfig.json`, and test config
2. Move core entity types first (`Vertex`, `Edge`, `Entities`, branded IDs, `EntityProperties`) — these have the fewest dependencies
3. Move the `Explorer` interface and all request/response types
4. Move shared utilities (`query`, `escapeString`, `batchPromisesSerially`, `NetworkError`, constants)
5. Move query templates and response mappers for each language
6. Move explorer factory functions, introducing the transport abstraction
7. Update `graph-explorer` to import from the new package
8. Update `@shared/types` — `ConnectionConfig` and `QueryEngine` should move to the new library since they define the connection contract

## Open Questions

- Package name: `graph-explorer-client`, `graph-query-client`, or something else?
- Should the `ConnectionConfig` type live in this new library or remain in `@shared/types`?
- Should the library include a default `fetch`-based transport, or should transport always be injected?
- How should the library handle the `uuid` dependency currently used for `queryId` generation in the explorers?

Contributor guide

Open the contributing guide

Research direction

Start by reading packages/graph-explorer/src/connector/ and the Explorer interface in useGEFetchTypes.ts, then trace the query templates, mappers, transport, and React integration boundaries listed in the issue. Done means a new packages/graph-explorer-client package owns the framework-independent API and graph query logic, while graph-explorer retains TanStack Query, Jotai, hooks, and display formatting without UI dependencies in the client.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
api, backend-api-design
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.