gchq / gchq/sleeper

Sleeper in general doesn't work with JDK 21+

Open
#6,851 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

### Description / Background

Sleeper becomes unusable after JDK20.

### Steps to reproduce

1. Build Sleeper on any JDK < 20.
2. Switch to JDK 21+.
3. Run scripts/utility/query.sh and try to query anything.

The following exception is produced:

```
Exception in thread "main" sleeper.core.statestore.StateStoreException: Failed updating state from transactions
at sleeper.core.statestore.transactionlog.TransactionLogHead.update(TransactionLogHead.java:229)
at sleeper.core.statestore.transactionlog.TransactionLogHead.update(TransactionLogHead.java:213)
at sleeper.core.statestore.transactionlog.TransactionLogPartitionStore.partitions(TransactionLogPartitionStore.java:86)
at sleeper.core.statestore.transactionlog.TransactionLogPartitionStore.getAllPartitions(TransactionLogPartitionStore.java:44)
at sleeper.core.statestore.DelegatingStateStore.getAllPartitions(DelegatingStateStore.java:69)
at sleeper.query.core.rowretrieval.QueryPlanner.init(QueryPlanner.java:127)
at sleeper.query.core.rowretrieval.QueryPlanner.init(QueryPlanner.java:91)
at sleeper.query.core.rowretrieval.QueryPlanner.initialiseNow(QueryPlanner.java:76)
at sleeper.clients.query.QueryClient.init(QueryClient.java:90)
at sleeper.clients.query.QueryCommandLineClient.run(QueryCommandLineClient.java:89)
at sleeper.clients.query.QueryClient.main(QueryClient.java:146)
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to unmarshall response (sun.misc.Unsafe or java.nio.DirectByteBuffer.(long, int) not available). Response Code: 200, Response Text: OK (SDK Attempt Count: 1)
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:130)
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:95)
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:73)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$0(BaseSyncClientHandler.java:66)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:60)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:52)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:60)
at software.amazon.awssdk.services.s3.DefaultS3Client.getObject(DefaultS3Client.java:6841)
at software.amazon.awssdk.services.s3.S3Client.getObject(S3Client.java:11196)
at sleeper.statestore.StateStoreArrowFileReadStore.loadPartitions(StateStoreArrowFileReadStore.java:108)
at sleeper.statestore.StateStoreArrowFileReadStore.loadSnapshot(StateStoreArrowFileReadStore.java:70)
at java.base/java.util.Optional.map(Optional.java:260)
at sleeper.statestore.transactionlog.snapshots.DynamoDBTransactionLogSnapshotLoader.loadLatestSnapshotInRange(DynamoDBTransactionLogSnapshotLoader.java:45)
at sleeper.core.statestore.transactionlog.TransactionLogHead.lambda$loadSnapshotIfNeeded$0(TransactionLogHead.java:241)
at java.base/java.util.Optional.flatMap(Optional.java:289)
at sleeper.core.statestore.transactionlog.TransactionLogHead.loadSnapshotIfNeeded(TransactionLogHead.java:241)
at sleeper.core.statestore.transactionlog.TransactionLogHead.update(TransactionLogHead.java:226)
... 10 more
Caused by: software.amazon.awssdk.core.exception.NonRetryableException: sun.misc.Unsafe or java.nio.DirectByteBuffer.(long, int) not available
at software.amazon.awssdk.core.exception.NonRetryableException$BuilderImpl.build(NonRetryableException.java:118)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler$HttpResponseHandlerAdapter.transformResponse(BaseSyncClientHandler.java:232)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler$HttpResponseHandlerAdapter.handle(BaseSyncClientHandler.java:213)
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleSuccessResponse(CombinedResponseHandler.java:98)
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:74)
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:59)
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:40)
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:52)
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:74)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:43)
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:79)
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:41)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:58)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:41)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.executeRequest(RetryableStage.java:93)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:56)
... 41 more
Caused by: java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.(long, int) not available
at org.apache.arrow.memory.util.MemoryUtil.directBuffer(MemoryUtil.java:174)
at org.apache.arrow.memory.ArrowBuf.getDirectBuffer(ArrowBuf.java:229)
at org.apache.arrow.memory.ArrowBuf.nioBuffer(ArrowBuf.java:224)
at org.apache.arrow.vector.ipc.ReadChannel.readFully(ReadChannel.java:87)
at org.apache.arrow.vector.ipc.message.MessageSerializer.readMessageBody(MessageSerializer.java:728)
at org.apache.arrow.vector.ipc.message.MessageChannelReader.readNext(MessageChannelReader.java:67)
at org.apache.arrow.vector.ipc.ArrowStreamReader.loadNextBatch(ArrowStreamReader.java:145)
at sleeper.statestore.StateStorePartitionsArrowFormat.read(StateStorePartitionsArrowFormat.java:163)
at sleeper.statestore.StateStoreArrowFileReadStore.lambda$loadPartitions$3(StateStoreArrowFileReadStore.java:114)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler$HttpResponseHandlerAdapter.transformResponse(BaseSyncClientHandler.java:225)
... 57 more
```

As this exception originates in Sleeper state store processing, this is very likely to occur in many components of Sleeper, essentially making the project unusable on modern JDKs.

### Technical Notes / Implementation Details

According to (https://github.com/apache/arrow/pull/36370) this is down to a change in argument types of a package private class DirectByteBuffer in JDK 21.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.