JanusGraph / JanusGraph/janusgraph
Janus graph failing silently on failing to remove a composite index and not respecting the maximum BigTable row size on updating a value of a Composite index.
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 5.8k
- Forks
- 1.2k
- Avg merge
- 13h 53m
- Merged PRs (30d)
- 6
Description
- Version:0.6.0
- Storage Backend: BigTable
- Mixed Index Backend: elasticsearch
- Link to discussed bug: https://github.com/JanusGraph/janusgraph/discussions/3257#discussioncomment-3942498
- Expected Behavior:
- Janus should not fail silently when failing to remove a composite index and should respect the maximum BigTable row size
- Current Behavior:
- Janus graph is failing silently on failing to remove a composite index and not respecting the maximum BigTable row size on updating a value of a Composite index.
- Steps to Reproduce:
- Create a composite index on a property/properties whose values are equal for lots of vertices in the graph. then try to remove it.
### Stack Trace
```
6898168 [grpc-default-executor-150] ERROR com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.async.AbstractRetryingOperation - Could not complete RPC. Failure #0, got: Status{code=FAILED_PRECONDITION, description=Error while reading table 'projects/integration-618527513/instances/curago0/tables/janusgraph' : Read returned 284MiB from row '$=\350Q6\211\240findin\347' which exceeds the limit of 256MiB. Make sure you are setting an appropriate request filter to retrieve only recent versions and only the columns you want. If columns are accumulating more versions than you need to read, you can also create a garbage collection policy: https://cloud.google.com/bigtable/docs/configuring-garbage-collection#versions, cause=null} on channel {cbt:99,grpc:ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=227, target=bigtable.googleapis.com:443}}}.
Trailers: Metadata(grpc-server-stats-bin=AAD+46OwFwAAAA,x-goog-ext-425905942-bin=Cg5ldXJvcGUtd2VzdDItYhIGY3VyYWdv,bigtable-channel-id={cbt:99,grpc:ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=227, target=bigtable.googleapis.com:443}}})
6898171 [data-puller-0] ERROR org.janusgraph.diskstorage.keycolumnvalue.scan.MultiThreadsRowsCollector - Could not load data from storage
java.io.UncheckedIOException: com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.io.IOExceptionWithStatus: Error in response stream
at org.apache.hadoop.hbase.client.ResultScanner$1.hasNext(ResultScanner.java:55)
at com.google.common.collect.Iterators$5.computeNext(Iterators.java:635)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
at org.janusgraph.diskstorage.hbase.HBaseKeyColumnValueStore$RowIterator.hasNext(HBaseKeyColumnValueStore.java:330)
at org.janusgraph.diskstorage.util.MetricInstrumentedStore.runWithMetrics(MetricInstrumentedStore.java:272)
at org.janusgraph.diskstorage.util.MetricInstrumentedIterator.hasNext(MetricInstrumentedIterator.java:73)
at org.janusgraph.diskstorage.keycolumnvalue.scan.MultiThreadsRowsCollector$DataPuller.run(MultiThreadsRowsCollector.java:221)
Caused by: com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.io.IOExceptionWithStatus: Error in response stream
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.scanner.ResultQueueEntry$ExceptionResultQueueEntry.getResponseOrThrow(ResultQueueEntry.java:100)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.scanner.ResponseQueueReader.getNextMergedRow(ResponseQueueReader.java:100)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.scanner.ResponseQueueReader.getNextMergedRow(ResponseQueueReader.java:106)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.scanner.ResumingStreamingResultScanner.next(ResumingStreamingResultScanner.java:78)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.scanner.ResumingStreamingResultScanner.next(ResumingStreamingResultScanner.java:33)
at com.google.cloud.bigtable.hbase.adapters.read.BigtableResultScannerAdapter$1.next(BigtableResultScannerAdapter.java:63)
at org.apache.hadoop.hbase.client.ResultScanner$1.hasNext(ResultScanner.java:53)
... 7 more
Caused by: com.google.bigtable.repackaged.io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Error while reading table 'projects/integration-618527513/instances/curago0/tables/janusgraph' : Read returned 284MiB from row '$=\350Q6\211\240findin\347' which exceeds the limit of 256MiB. Make sure you are setting an appropriate request filter to retrieve only recent versions and only the columns you want. If columns are accumulating more versions than you need to read, you can also create a garbage collection policy: https://cloud.google.com/bigtable/docs/configuring-garbage-collection#versions
at com.google.bigtable.repackaged.io.grpc.Status.asRuntimeException(Status.java:524)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.async.AbstractRetryingOperation.onError(AbstractRetryingOperation.java:215)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.async.AbstractRetryingOperation.onClose(AbstractRetryingOperation.java:178)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.scanner.RetryingReadRowsOperation.onClose(RetryingReadRowsOperation.java:231)
at com.google.bigtable.repackaged.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at com.google.bigtable.repackaged.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at com.google.bigtable.repackaged.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.io.RefreshingOAuth2CredentialsInterceptor$UnAuthResponseListener.onClose(RefreshingOAuth2CredentialsInterceptor.java:82)
at com.google.bigtable.repackaged.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at com.google.bigtable.repackaged.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at com.google.bigtable.repackaged.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.io.Watchdog$WatchedCall$1.onClose(Watchdog.java:188)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.io.ChannelPool$InstrumentedChannel$2.onClose(ChannelPool.java:213)
at com.google.bigtable.repackaged.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at com.google.bigtable.repackaged.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at com.google.bigtable.repackaged.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at com.google.bigtable.repackaged.io.grpc.census.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:701)
at com.google.bigtable.repackaged.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at com.google.bigtable.repackaged.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at com.google.bigtable.repackaged.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at com.google.bigtable.repackaged.io.grpc.census.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:398)
at com.google.bigtable.repackaged.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:617)
at com.google.bigtable.repackaged.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
at com.google.bigtable.repackaged.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:803)
at com.google.bigtable.repackaged.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:782)
at com.google.bigtable.repackaged.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at com.google.bigtable.repackaged.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
6898193 [Thread-13] INFO org.janusgraph.graphdb.olap.job.IndexRemoveJob - Index byEntityTypeAndTask metrics: success-tx: 1 doc-updates: 0 succeeded: 0
6898283 [Thread-11] INFO org.janusgraph.graphdb.olap.job.IndexRemoveJob - Index byEntityTypeAndTask metrics: success-tx: 2 doc-updates: 0 succeeded: 0
6898284 [Thread-11] INFO org.janusgraph.graphdb.database.management.ManagementSystem - Index update job successful for [byEntityTypeAndTask]
```
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the composite-index removal flow around org.janusgraph.graphdb.olap.job.IndexRemoveJob and ManagementSystem, then trace storage failures through HBaseKeyColumnValueStore and MultiThreadsRowsCollector. Reproduce the Bigtable case with a high-cardinality composite index and an oversized row. Done means removal failures are surfaced instead of reported as successful, and composite-index updates respect Bigtable's row-size limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, google-cloud, java
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100