[3.2.2][Gremlin] Relationship discovery fails with MemoryLimitExceededException for high-cardinality edge labels

Description

Graph Explorer 3.2.2 fails to discover edge connections for a Neptune property graph with only three edge labels, two of which have several million edges.

The graph synchronized successfully in Graph Explorer previously, even when it contained more data. The current PG-Gremlin synchronization retrieves the node and edge counts, but relationship discovery fails.

A PG-openCypher connection to the same Neptune database completes relationship discovery successfully.

Environment

• Graph Explorer version: 3.2.2

• Deployment: Amazon SageMaker notebook

• Database: Amazon Neptune Serverless

• Query language: PG-Gremlin

• Serverless capacity range:

  • Minimum: 1 NCU

  • Maximum: 128 NCUs

• Total edges: 19,928,805

• Number of edge labels: 3

The edge-label names below have been anonymized:

Edge label | Count
-- | --
LabelA | 209,968
LabelB | 10,972,096
LabelC | 8,746,741

Steps to reproduce

1. Connect Graph Explorer 3.2.2 to the Neptune database using PG-Gremlin.

2. Synchronize or refresh the connection.

3. Wait for relationship discovery to run.

4. Observe “Relationship Discovery Failed.”

Graph Explorer submits a query with this shape:

g.E().hasLabel('LabelA', 'LabelB', 'LabelC')

  .group()

    .by(label())

    .by(

      limit(10000)

        .project('sourceType', 'targetType')

        .by(outV().label())

        .by(inV().label())

        .dedup()

        .fold()

    )

After approximately 30 seconds, Neptune returns HTTP 500:

{

  "code": "MemoryLimitExceededException",

  "detailedMessage": "Query cannot be completed due to memory limitations.",

  "message": "Query cannot be completed due to memory limitations."

}

Individual per-label query succeeds

Running the equivalent discovery query for one label without group() succeeds:

g.E().hasLabel('LabelA')

  .limit(10000)

  .project('sourceType', 'targetType')

    .by(outV().label())

    .by(inV().label())

  .dedup()

Example result:

sourceType: Company

targetType: Address

The individual edge-count queries also complete successfully:

g.E().hasLabel('LabelA').count()

g.E().hasLabel('LabelB').count()

g.E().hasLabel('LabelC').count()

Expected behavior

Relationship discovery should complete using bounded sampling for each edge label.

For a graph with only three edge labels, Graph Explorer could issue three separately limited queries and combine their results client-side.

Actual behavior

The batched Gremlin query uses group() before the per-label limit(10000) can bound the initial work. It therefore enumerates and groups all 19,928,805 matching edges and eventually fails with MemoryLimitExceededException.

This prevents the PG-Gremlin connection from completing relationship discovery.

Regression and related change

This appears related to the Gremlin edge-connection batching introduced in Graph Explorer 3.2.2:

https://github.com/aws/graph-explorer/pull/2100

The pull request documents that the Gremlin group() query enumerates every edge of the batched types and that the inner limit does not cap the initial raw scan.

For graphs with thousands of edge labels, batching reduces the number of HTTP requests. For this graph, however, batching three high-cardinality labels saves only two requests while changing the work into a grouping operation over nearly 20 million edges.

Workarounds confirmed

• PG-openCypher relationship discovery succeeds against the same Neptune graph.

• Individual early-limited PG-Gremlin queries succeed.

• Ordinary Gremlin queries continue to work; the failure is specific to relationship discovery.

Suggested fixes

Any of the following would address the problem:

1. Use per-label Gremlin discovery when the graph contains only a small number of edge labels.

2. When a batched request returns MemoryLimitExceededException or times out, retry the labels individually.

3. Make the Gremlin relationship-discovery strategy or batch size configurable.

4. Detect high-cardinality edge types and avoid the aggregated group() query.