cube-js / cube-js/cube

CubeStore v1.6.67: DataFusion panic in optimize_projections / DFSchema::qualified_field ("index out of bounds: the len is 3 but the index is 3") on rolling-window rollup query

Open
#11,647 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
20.8k
Forks
2.1k
Avg merge
1d 2h
Merged PRs (30d)
181

Description

## Describe the bug

CubeStore's router crashes a query with a Rust panic in DataFusion's `optimize_projections` pass, immediately and repeatedly, when serving a rollup query for a cube whose only measures are a `count_distinct_approx` rolling-window measure, a `sum` rolling-window measure, and a `number`-type measure computed as a ratio of the two (see schema below). The router process itself does not crash-loop — the panic happens inside a `tokio::runtime::blocking::pool` task and appears to be caught per-query — but the query fails every time, and we saw hundreds of occurrences within the first ~40 minutes of rolling this version out.

## To Reproduce

We don't have the exact generated SQL captured (our CubeStore log level was `info`, which doesn't log per-query SQL text for this code path), only the schema/measure shape that consistently triggers it. Relevant cube definition (names genericized, shape unchanged):

```yaml
cubes:
- name: events
data_source: default
sql_table: my_schema.my_table

dimensions:
- name: group_id
sql: group_id
type: number
- name: entity_id
sql: entity_id
type: number
- name: event_time
sql: event_time
type: time

measures:
- name: count_trailing_1_hr
type: count_distinct_approx
sql: id_a || '-' || TO_CHAR(entity_id)
rolling_window:
trailing: 1 hour

- name: sum_trailing_1_hr
type: sum
sql: value
rolling_window:
trailing: 1 hour

- name: avg_trailing_1_hr
type: number
sql: "{CUBE.sum_trailing_1_hr} / {CUBE.count_trailing_1_hr}"

pre_aggregations:
- name: avg_by_minute_pre_agg
type: rollup
measures:
- CUBE.sum_trailing_1_hr
- CUBE.count_trailing_1_hr
dimensions:
- CUBE.group_id
- CUBE.entity_id
time_dimension: CUBE.event_time
granularity: minute
partition_granularity: hour
```

Querying `count_trailing_1_hr`, `avg_trailing_1_hr` (the computed ratio measure) together, filtered by `group_id`, against this pre-aggregation reliably triggers the panic in our prod environment. The pre-aggregation's own group-by/unique-key column set has exactly 3 columns (`group_id`, `entity_id`, and the truncated time bucket), which may or may not be coincidental to the panic's "len is 3".

We were **not** able to reproduce this in two separate targeted tests before rolling out to production:
- A single-instance correctness check against an isolated, freshly-built CubeStore v1.6.67 pair.
- A burst of 30 concurrent identical queries against a real (non-isolated) staging CubeStore v1.6.67 cluster.

It only appeared once real production traffic hit it, so we suspect it's tied to a specific data condition or query parameter combination we haven't isolated (rather than pure concurrency).

## Expected behavior

Query resolves normally (as it does, unmodified, on CubeStore v1.5.3).

## Panic / stack trace

```
thread 'cubestore-main' panicked at /usr/local/cargo/git/checkouts/arrow-datafusion-9108e463a07fefed/ea20497/datafusion/common/src/dfschema.rs:345:31:
index out of bounds: the len is 3 but the index is 3
stack backtrace:
0: __rustc::rust_begin_unwind
1: core::panicking::panic_fmt
2: core::panicking::panic_bounds_check
3: datafusion_common::dfschema::DFSchema::qualified_field
4: as alloc::vec::spec_from_iter::SpecFromIter>::from_iter
5: >::map_elements
6: as datafusion_common::tree_node::TreeNodeContainer>::map_elements
7: datafusion_expr::logical_plan::tree_node::::map_children
8: datafusion_optimizer::optimize_projections::optimize_projections::{{closure}}
9: datafusion_optimizer::optimize_projections::optimize_projections
10: core::ops::function::impls:: for &mut F>::call_mut
11: as datafusion_common::tree_node::TreeNodeContainer>::map_elements
12: <(C0,C1) as datafusion_common::tree_node::TreeNodeContainer>::map_elements
...
43: tokio::runtime::blocking::pool::Inner::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
```

This occurred 316 times between 2026-08-25T14:59:53Z and at least 2026-08-25T15:36:04Z, roughly every few seconds, until we rolled back.

## Versions

- Cube / CubeStore: `v1.6.67`
- Vendored DataFusion commit (from the panic path): `arrow-datafusion-9108e463a07fefed/ea20497`
- Deployment: Kubernetes (AKS), CubeStore router + worker as separate StatefulSets, `CUBEJS_ROLLUP_ONLY=true` on the API tier
- Previously working on: CubeStore `v1.5.3` (no issue)

## Additional context

This surfaced during an upgrade from v1.5.3 to v1.6.67 that was otherwise validated for a separate, known concurrency-related regression in this same cube's rolling-window query (unrelated crash-loop, already understood and load-tested for). This `dfschema.rs` panic is a second, distinct issue found only after promoting to production, and we rolled back to v1.5.3 to mitigate. Happy to provide more detail (full un-truncated backtrace, `RUST_BACKTRACE=full` capture, more of the schema) if useful.

Contributor guide

Open the contributing guide

Research direction

Start with the panic path in DataFusion's optimize_projections and DFSchema::qualified_field, using the vendored commit ea20497 and the v1.6.67 versus v1.5.3 behavior as comparison points. Reconstruct the CubeStore rolling-window query from the supplied cube schema, then isolate the data or query condition that produces the three-column mismatch. Done means a reproducible regression case and a query that resolves without a panic.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, rust, sql
Domain
backend, databases, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.