JanusGraph / JanusGraph/janusgraph

High Memory Pressure In RestElasticSearchClient

Open
#4,684 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
5.8k
Forks
1.2k
Avg merge
13h 53m
Merged PRs (30d)
6

Description

**Describe the feature:**
In [RestElasticSearchClient](https://github.com/JanusGraph/janusgraph/blob/master/janusgraph-es/src/main/java/org/janusgraph/diskstorage/es/rest/RestElasticSearchClient.java) there are a few places that can cause a high amount of memory pressure unnecessarily.

1. The [submitted mutations](https://github.com/JanusGraph/janusgraph/blob/4a576f67ff0e53b699ed078dad964d448bc94b10/janusgraph-es/src/main/java/org/janusgraph/diskstorage/es/rest/RestElasticSearchClient.java#L497-L501) are passed in via a collection, which means a reference to the collection is still valid at the call site. Once the mutations are serialized however those unserialized forms are no longer necessary to keep around. The unserialized mutations could be released from the passed in collection so they're no longer prevented from being GC'ed. The collection itself though may stick around.
2. The [serialized mutations](https://github.com/JanusGraph/janusgraph/blob/4a576f67ff0e53b699ed078dad964d448bc94b10/janusgraph-es/src/main/java/org/janusgraph/diskstorage/es/rest/RestElasticSearchClient.java#L493) are presented to the client by reading them off an iterator. This means there's a reference to the collection of all serialized mutations so long as that iterator lives, and even though things have been successfully sent to ElasticSearch the collection prevents the successfully submitted requests from being garbage collected.
3. The [bulking of the serialized requests](https://github.com/JanusGraph/janusgraph/blob/4a576f67ff0e53b699ed078dad964d448bc94b10/janusgraph-es/src/main/java/org/janusgraph/diskstorage/es/rest/RestElasticSearchClient.java#L437-L461) can cause unnecessary byte arrays to be allocated as the `ByteArrayOutputStream` grows its underlying `byte []`. These may then be released as the array grows, but then another allocation, usually 2x the previous one, is done which may itself be transitory. However the [final step](https://github.com/JanusGraph/janusgraph/blob/4a576f67ff0e53b699ed078dad964d448bc94b10/janusgraph-es/src/main/java/org/janusgraph/diskstorage/es/rest/RestElasticSearchClient.java#L461) causes a copy of the final state of the `byte[]` within the `ByteArrayOutputStream`, which can be avoided.

As a reminder the individual elements of each mutation batch can reach 100MB by default. The current client logic will split up each element into chunks so that they're below that limit, but a chunk of 1 is valid. So the copying here may reach non-trivial amounts, especially against concurrent client requests. So anything that can be done to release memory around these mutations as soon as possible maximizes the space JanusGraph can work with if the garbage collector starts getting desperate.

For what it's worth I've had a few OOM's occur in around the present `writeTo` logic:

```
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.Arrays.copyOf(Unknown Source) ~[?:?]
at java.base/java.io.ByteArrayOutputStream.grow(Unknown Source) ~[?:?]
at java.base/java.io.ByteArrayOutputStream.ensureCapacity(Unknown Source) ~[?:?]
at java.base/java.io.ByteArrayOutputStream.write(Unknown Source) ~[?:?]
at java.base/java.io.OutputStream.write(Unknown Source) ~[?:?]
at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient$RequestBytes.writeTo(RestElasticSearchClient.java:441) ~[janusgraph-es-1.1.0-20240628-052016.a3393a1.jar:?]
at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient$RequestBytes.access$500(RestElasticSearchClient.java:398) ~[janusgraph-es-1.1.0-20240628-052016.a3393a1.jar:?]
at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.buildBulkRequestInput(RestElasticSearchClient.java:450) ~[janusgraph-es-1.1.0-20240628-052016.a3393a1.jar:?]
at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:556) ~[janusgraph-es-1.1.0-20240628-052016.a3393a1.jar:?]
at org.janusgraph.diskstorage.es.ElasticSearchIndex.mutate(ElasticSearchIndex.java:909) ~[janusgraph-es-1.1.0-20240628-052016.a3393a1.jar:?]
at org.janusgraph.diskstorage.util.MetricInstrumentedIndexProvider.lambda$mutate$0(MetricInstrumentedIndexProvider.java:68) ~[janusgraph-core-1.1.0-20240628-052016.a3393a1.jar:?]
at org.janusgraph.diskstorage.util.MetricInstrumentedIndexProvider$$Lambda$1408/0x0000000840a53840.run(Unknown Source) ~[?:?]
at org.janusgraph.diskstorage.util.MetricInstrumentedIndexProvider.runWithMetrics(MetricInstrumentedIndexProvider.java:153) ~[janusgraph-core-1.1.0-20240628-052016.a3393a1.jar:?]
at org.janusgraph.diskstorage.util.MetricInstrumentedIndexProvider.mutate(MetricInstrumentedIndexProvider.java:68) ~[janusgraph-core-1.1.0-20240628-052016.a3393a1.jar:?]
at org.janusgraph.diskstorage.indexing.IndexTransaction$1.call(IndexTransaction.java:151) ~[janusgraph-core-1.1.0-20240628-052016.a3393a1.jar:?]
at org.janusgraph.diskstorage.indexing.IndexTransaction$1.call(IndexTransaction.java:148) ~[janusgraph-core-1.1.0-20240628-052016.a3393a1.jar:?]
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:66) ~[janusgraph-core-1.1.0-20240628-052016.a3393a1.jar:?]
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:52) ~[janusgraph-core-1.1.0-20240628-052016.a3393a1.jar:?]
at org.janusgraph.diskstorage.indexing.IndexTransaction.flushInternal(IndexTransaction.java:148) ~[janusgraph-core-1.1.0-20240628-052016.a3393a1.jar:?]
at org.janusgraph.diskstorage.indexing.IndexTransaction.commit(IndexTransaction.java:129) ~[janusgraph-core-1.1.0-20240628-052016.a3393a1.jar:?]
at org.janusgraph.diskstorage.BackendTransaction.commitIndexes(BackendTransaction.java:156) ~[janusgraph-core-1.1.0-20240628-052016.a3393a1.jar:?]
at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:1023) [janusgraph-core-1.1.0-20240628-052016.a3393a1.jar:?]
at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1604) [janusgraph-core-1.1.0-20240628-052016.a3393a1.jar:?]
at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.doCommit(JanusGraphBlueprintsGraph.java:322) [janusgraph-core-1.1.0-20240628-052016.a3393a1.jar:?]
at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:104) [gremlin-core-3.7.2.jar:3.7.2]
at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.commit(JanusGraphBlueprintsGraph.java:300) [janusgraph-core-1.1.0-20240628-052016.a3393a1.jar:?]
at org.janusgraph.graphdb.management.JanusGraphManager.lambda$commitAll$2(JanusGraphManager.java:201) [janusgraph-core-1.1.0-20240628-052016.a3393a1.jar:?]
at org.janusgraph.graphdb.management.JanusGraphManager$$Lambda$1061/0x00000008408e1440.accept(Unknown Source) [janusgraph-core-1.1.0-20240628-052016.a3393a1.jar:?]
at java.base/java.util.concurrent.ConcurrentHashMap.forEach(Unknown Source) ~[?:?]
at org.janusgraph.graphdb.management.JanusGraphManager.commitAll(JanusGraphManager.java:199) ~[janusgraph-core-1.1.0-20240628-052016.a3393a1.jar:?]
at org.apache.tinkerpop.gremlin.server.handler.AbstractSession.closeTransaction(AbstractSession.java:845) ~[gremlin-server-3.7.2.jar:3.7.2]
at org.apache.tinkerpop.gremlin.server.handler.AbstractSession.handleIterator(AbstractSession.java:545) ~[gremlin-server-3.7.2.jar:3.7.2]
```

**Describe a specific use case for the feature:**
[If possible add a description of the specific use case.]

Contributor guide

Open the contributing guide

Research direction

Start in janusgraph-es/src/main/java/org/janusgraph/diskstorage/es/rest/RestElasticSearchClient.java, especially the bulking lines 437-461, serialized mutation handling near 493-501, and bulkRequest around line 556. Trace writeTo and buildBulkRequestInput against the reported stack trace; done means reducing unnecessary mutation and byte-array retention without changing the Elasticsearch bulk behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, java
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.