cubestore reports memory chunk with owner node is trying to be repartitioned or compacted on non owner node
- Dominant language
- Rust
- Stars
- 20.8k
- Forks
- 2.1k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 181
Description
**Problem**
I am using a KSQL cube schema. My deployment is to a Kubernetes cluster and I have 1 cubestore-router and 4 cubestore-workers.
Occasionally, I see this error in the "cubestore-router", and the related query from the "cube-api" fails with an error:
```
2023-10-04 16:11:59,032 ERROR [cubestore::http] Error processing HTTP command: Internal: Execution error: Internal: In memory chunk IdRow { id: 474, row: Chunk { partition_id: 68, row_co
unt: 1, uploaded: true, active: true, last_used: None, in_memory: true, created_at: Some(2023-10-04T16:10:53.310138803Z), oldest_insert_at: Some(2023-10-04T16:10:53.310138989Z), deactivated_at:
None, suffix: Some("cmbwqeed"), file_size: None, replay_handle_id: Some(393), min: Some(Row { values: [Int(-2372103810349568798), Timestamp(TimestampValue { unix_nano: 1696434712007000000, str:
"2023-10-04T15:51:52.007Z" }), String("{}"), String("[]"), .....omitted for brevity.... } with owner node 'dbg-cubestor
e-worker-0.dbg-cubestore-worker-headless:9001' is trying to be repartitioned or compacted on non owner node 'dbg-cubestore-worker-1.dbg-cubestore-worker-headless:9001'
0: >::from
1: as core::future::future::Future>::poll
2: as core::future::future::Future>::poll
3: as core::future::future::Future>::poll
4: as core::future::future::Future>::poll
5: as core::future::future::Future>::poll
6: tokio::runtime::task::core::CoreStage::poll
7: tokio::runtime::task::harness::poll_future
8: tokio::runtime::task::harness::Harness::poll
9: std::thread::local::LocalKey::with
10: tokio::runtime::thread_pool::worker::Context::run_task
11: tokio::runtime::thread_pool::worker::Context::run
12: tokio::macros::scoped_tls::ScopedKey::set
13: tokio::runtime::thread_pool::worker::run
14: as core::future::future::Future>::poll
15: tokio::runtime::task::harness::Harness::poll
16: tokio::runtime::blocking::pool::Inner::run
17: std::sys_common::backtrace::__rust_begin_short_backtrace
18: core::ops::function::FnOnce::call_once{{vtable.shim}}
19: as core::ops::function::FnOnce>::call_once
at /rustc/dc80ca78b6ec2b6bba02560470347433bcd0bb3c/library/alloc/src/[boxed.rs:1951:9](http://boxed.rs:1951:9/)
20: as core::ops::function::FnOnce>::call_once
at /rustc/dc80ca78b6ec2b6bba02560470347433bcd0bb3c/library/alloc/src/[boxed.rs:1951:9](http://boxed.rs:1951:9/)
21: std::sys::unix::thread::Thread::new::thread_start
at /rustc/dc80ca78b6ec2b6bba02560470347433bcd0bb3c/library/std/src/sys/unix/[thread.rs:108:17](http://thread.rs:108:17/)
22: start_thread
23: clone
```
**I'm hoping that you can explain to me the situation where this error can happen?** FWIW, I see the error in the cubestore-router log. I also see a related error in the "cube-api" node, which I attribute to this error floating up to where the query was issued.
cubestore-router settings:
```
CUBESTORE_SERVER_NAME=dbg-cubestore-router:9999
CUBESTORE_PORT=3306
CUBESTORE_HTTP_PORT=3030
CUBESTORE_META_PORT=9999
CUBESTORE_STATUS_PORT=3031
CUBESTORE_WORKERS=dbg-cubestore-worker-0.dbg-cubestore-worker-headless:9001,dbg-cubestore-worker-1.dbg-cubestore-worker-headless:9001,dbg-cubestore-worker-2.dbg-cubestore-worker-headless:9001,dbg-cubestore-worker-3.dbg-cubestore-worker-headless:9001
CUBESTORE_DATA_DIR=/cube/.cubestore/data
CUBESTORE_REMOTE_DIR=/cube/data
CUBESTORE_LOG_LEVEL=trace
```
each cubestore-worker settings:
```
CUBESTORE_SERVER_NAME=dbg-cubestore-worker-0.dbg-cubestore-worker-headless:9001
CUBESTORE_WORKER_PORT=9001
CUBESTORE_WORKERS=dbg-cubestore-worker-0.dbg-cubestore-worker-headless:9001,dbg-cubestore-worker-1.dbg-cubestore-worker-headless:9001,dbg-cubestore-worker-2.dbg-cubestore-worker-headless:9001,dbg-cubestore-worker-3.dbg-cubestore-worker-headless:9001
CUBESTORE_META_ADDR=dbg-cubestore-router:9999
CUBESTORE_DATA_DIR=/cube/.cubestore/data
CUBESTORE_REMOTE_DIR=/cube/data
CUBESTORE_STALE_STREAM_TIMEOUT=30
CUBESTORE_LOG_LEVEL=trace
```
The "cubestore-worker" server names are 0, 1, 2, or 3 as expected for each worker.
CUBESTORE_REMOTE_DIR is a shared drive, CUBESTORE_DATA_DIR is not.
Thanks
Contributor guide
Assessment
This issue has not been assessed yet.