Error when writing to fluss: IndexOutOfBoundsException
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/alibaba/fluss/issues) and found nothing similar.
### Fluss version
0.6.0 (latest release)
### Please describe the bug 🐞
While writing a large amount of data(for example, after writing 10 million+ records to the flux table, consume this table and write to another table) into fluss, the client receives the following error
> 2025-03-26 01:51:46,314 ERROR com.alibaba.fluss.server.replica.ReplicaManager [] - Error put records to local kv on replica TableBucket{tableId=43, bucket=0}
java.lang.IndexOutOfBoundsException: capacity: 262144, index: 262144, put length: 1
at com.alibaba.fluss.memory.MemorySegment.put(MemorySegment.java:324) ~[fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.record.ChangeTypeVectorWriter.writeChangeType(ChangeTypeVectorWriter.java:43) ~[fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.record.MemoryLogRecordsArrowBuilder.append(MemoryLogRecordsArrowBuilder.java:169) ~[fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.server.kv.wal.ArrowWalBuilder.append(ArrowWalBuilder.java:47) ~[fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.server.kv.KvTablet.lambda$putAsLeader$0(KvTablet.java:338) ~[fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.utils.concurrent.LockUtils.inLock(LockUtils.java:41) ~[fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.utils.concurrent.LockUtils.inWriteLock(LockUtils.java:64) ~[fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.server.kv.KvTablet.putAsLeader(KvTablet.java:271) ~[fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.server.replica.Replica.lambda$putRecordsToLeader$10(Replica.java:847) ~[fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.utils.concurrent.LockUtils.inLock(LockUtils.java:41) ~[fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.utils.concurrent.LockUtils.inReadLock(LockUtils.java:54) ~[fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.server.replica.Replica.putRecordsToLeader(Replica.java:833) ~[fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.server.replica.ReplicaManager.putToLocalKv(ReplicaManager.java:894) [fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.server.replica.ReplicaManager.putRecordsToKv(ReplicaManager.java:430) [fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.server.tablet.TabletService.putKv(TabletService.java:143) [fluss-server-0.6.0.jar:0.6.0]
at jdk.internal.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
at com.alibaba.fluss.rpc.netty.server.RequestProcessor.processRequest(RequestProcessor.java:108) [fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.rpc.netty.server.RequestProcessor.run(RequestProcessor.java:84) [fluss-server-0.6.0.jar:0.6.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
### Solution
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the stack trace in ChangeTypeVectorWriter.java, MemoryLogRecordsArrowBuilder.java, and ArrowWalBuilder.java, then trace the call from KvTablet.putAsLeader. Reproduce the large write-and-consume scenario described for 10 million or more records. Done means the workload no longer produces the IndexOutOfBoundsException when writing to the destination table.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100