array_agg panics on large rows.
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
```
let df = ctx.sql(r#"
SELECT count(payload) from demo
"#).await?;
+---------------------+
| COUNT(demo.payload) |
+---------------------+
| 68934 |
+---------------------+
```
Then
```
let df = ctx.sql(r#"
SELECT array_agg(payload) from demo
"#).await?;
```
After running for about 2 mins, panics
```
thread 'main' panicked at 'attempt to add with overflow', /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/comfy-table-6.1.4/src/utils/mod.rs:44:9
stack backtrace:
0: rust_begin_unwind
at /rustc/c18a5e8a5b1afb0d7a582fe9ebad4c1996c90da3/library/std/src/panicking.rs:575:5
1: core::panicking::panic_fmt
at /rustc/c18a5e8a5b1afb0d7a582fe9ebad4c1996c90da3/library/core/src/panicking.rs:64:14
2: core::panicking::panic
at /rustc/c18a5e8a5b1afb0d7a582fe9ebad4c1996c90da3/library/core/src/panicking.rs:114:5
3: comfy_table::utils::ColumnDisplayInfo::width
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/comfy-table-6.1.4/src/utils/mod.rs:44:9
4: comfy_table::utils::formatting::borders::draw_top_border
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/comfy-table-6.1.4/src/utils/formatting/borders.rs:54:40
5: comfy_table::utils::formatting::borders::draw_borders
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/comfy-table-6.1.4/src/utils/formatting/borders.rs:21:20
6: comfy_table::utils::build_table
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/comfy-table-6.1.4/src/utils/mod.rs:51:5
7: comfy_table::table::Table::lines
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/comfy-table-6.1.4/src/table.rs:95:9
8: ::fmt
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/comfy-table-6.1.4/src/table.rs:47:25
9: core::fmt::write
at /rustc/c18a5e8a5b1afb0d7a582fe9ebad4c1996c90da3/library/core/src/fmt/mod.rs:1230:17
10: std::io::Write::write_fmt
at /rustc/c18a5e8a5b1afb0d7a582fe9ebad4c1996c90da3/library/std/src/io/mod.rs:1682:15
11: <&std::io::stdio::Stdout as std::io::Write>::write_fmt
at /rustc/c18a5e8a5b1afb0d7a582fe9ebad4c1996c90da3/library/std/src/io/stdio.rs:715:9
12: ::write_fmt
at /rustc/c18a5e8a5b1afb0d7a582fe9ebad4c1996c90da3/library/std/src/io/stdio.rs:689:9
13: std::io::stdio::print_to
at /rustc/c18a5e8a5b1afb0d7a582fe9ebad4c1996c90da3/library/std/src/io/stdio.rs:1007:21
14: std::io::stdio::_print
at /rustc/c18a5e8a5b1afb0d7a582fe9ebad4c1996c90da3/library/std/src/io/stdio.rs:1074:5
15: arrow_cast::pretty::print_batches
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/arrow-cast-38.0.0/src/pretty.rs:62:5
16: datafusion::dataframe::DataFrame::show::{{closure}}
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/datafusion-24.0.0/src/dataframe.rs:682:12
17: query_playground::main::{{closure}}
at ./src/query_playground.rs:785:14
18: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/park.rs:283:63
19: tokio::runtime::coop::with_budget
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/coop.rs:102:5
20: tokio::runtime::coop::budget
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/coop.rs:68:5
21: tokio::runtime::park::CachedParkThread::block_on
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/park.rs:283:31
22: tokio::runtime::context::BlockingRegionGuard::block_on
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/context.rs:315:13
23: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/scheduler/multi_thread/mod.rs:66:9
24: tokio::runtime::runtime::Runtime::block_on
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/runtime.rs:284:45
25: query_playground::main
at ./src/query_playground.rs:812:5
26: core::ops::function::FnOnce::call_once
at /rustc/c18a5e8a5b1afb0d7a582fe9ebad4c1996c90da3/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
```
### To Reproduce
Create a table with a large number of rows and use array_agg on one of its columns
### Expected behavior
array_agg should not panic on large rows.
### Additional context
```
datafusion = { version = "24.0.0" }
```
Contributor guide
Research direction
Reproduce the large-row array_agg query described in the issue and inspect the reported path through DataFrame::show, arrow_cast::pretty::print_batches, and comfy-table formatting. Use src/query_playground.rs around lines 785 and 812 as the reproduction entry point; done means displaying the result no longer panics on large rows.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100