apache / apache/pulsar

Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1

Open
#12,284 18 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Describe the bug**
Following [this upgrade guide](https://pulsar.apache.org/docs/en/administration-upgrade/) to upgrade pulsar cluster node by node, from `2.7.0` to `2.8.1`.

Presto CLI version `332` is after the upgrade.

Trying to query in Presto CLI from one of the topics,
`apache-pulsar/bin/pulsar sql`
`select * from pulsar."public/default"."test";`
returns
`Query 20211006_093830_00017_59f2r failed: java.nio.BufferUnderflowException`
(See below for full logs)

However, this only affects some topics. Other topics in the same tenant and namespace can be queried just fine.

Also, using [`pulsar flink connector`](https://github.com/streamnative/pulsar-flink) and [`pulsar client python api`](https://pulsar.apache.org/docs/en/client-libraries-python/#reader-interface-example) to consume data from the problematic topics are working as expected, no issues.

Full logs
```
2021-10-06T09:38:30.837Z ERROR remote-task-callback-89 io.prestosql.execution.StageStateMachine Stage 20211006_093830_00017_59f2r.1 failed
com.google.common.util.concurrent.UncheckedExecutionException: java.nio.BufferUnderflowException
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2051)
at com.google.common.cache.LocalCache.get(LocalCache.java:3951)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4935)
at org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.getSchemaByVersion(PulsarSqlSchemaInfoProvider.java:76)
at org.apache.pulsar.sql.presto.PulsarRecordCursor.advanceNextPosition(PulsarRecordCursor.java:485)
at io.prestosql.spi.connector.RecordPageSource.getNextPage(RecordPageSource.java:90)
at io.prestosql.operator.TableScanOperator.getOutput(TableScanOperator.java:302)
at io.prestosql.operator.Driver.processInternal(Driver.java:379)
at io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283)
at io.prestosql.operator.Driver.tryWithLock(Driver.java:675)
at io.prestosql.operator.Driver.processFor(Driver.java:276)
at io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1075)
at io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
at io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484)
at io.prestosql.$gen.Presto_332__testversion____20211005_104810_2.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:510)
at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:427)
at org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.loadSchema(PulsarSqlSchemaInfoProvider.java:106)
at org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.access$000(PulsarSqlSchemaInfoProvider.java:49)
at org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider$1.load(PulsarSqlSchemaInfoProvider.java:61)
at org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider$1.load(PulsarSqlSchemaInfoProvider.java:58)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
... 18 more
```

PS: Saw similar [issue](https://github.com/apache/pulsar/pull/9631), but that one is about **byte schema data**, mine is **string schema data**.

Suspect presto has messed up its cache somehow. Any pointer on how to identify the root cause and/or fix this issue will be greatly appreciated. Thanks!

Contributor guide

Open the contributing guide

Research direction

Start with PulsarSqlSchemaInfoProvider.getSchemaByVersion and loadSchema, where the trace reports the BufferUnderflowException, then follow how PulsarRecordCursor.advanceNextPosition reads schema data. Reproduce the Presto CLI query against a failing string-schema topic and a working topic after the 2.7.0-to-2.8.1 upgrade; done when the affected query no longer fails.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
backend, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.