apache / apache/datafusion

array_agg panics on large rows.

Open
#6,524 13 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

```
let df = ctx.sql(r#"
SELECT count(payload) from demo
"#).await?;

+---------------------+
| COUNT(demo.payload) |
+---------------------+
| 68934 |
+---------------------+
```

Then

```
let df = ctx.sql(r#"
SELECT array_agg(payload) from demo
"#).await?;
```

After running for about 2 mins, panics

```
thread 'main' panicked at 'attempt to add with overflow', /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/comfy-table-6.1.4/src/utils/mod.rs:44:9
stack backtrace:
0: rust_begin_unwind
at /rustc/c18a5e8a5b1afb0d7a582fe9ebad4c1996c90da3/library/std/src/panicking.rs:575:5
1: core::panicking::panic_fmt
at /rustc/c18a5e8a5b1afb0d7a582fe9ebad4c1996c90da3/library/core/src/panicking.rs:64:14
2: core::panicking::panic
at /rustc/c18a5e8a5b1afb0d7a582fe9ebad4c1996c90da3/library/core/src/panicking.rs:114:5
3: comfy_table::utils::ColumnDisplayInfo::width
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/comfy-table-6.1.4/src/utils/mod.rs:44:9
4: comfy_table::utils::formatting::borders::draw_top_border
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/comfy-table-6.1.4/src/utils/formatting/borders.rs:54:40
5: comfy_table::utils::formatting::borders::draw_borders
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/comfy-table-6.1.4/src/utils/formatting/borders.rs:21:20
6: comfy_table::utils::build_table
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/comfy-table-6.1.4/src/utils/mod.rs:51:5
7: comfy_table::table::Table::lines
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/comfy-table-6.1.4/src/table.rs:95:9
8: ::fmt
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/comfy-table-6.1.4/src/table.rs:47:25
9: core::fmt::write
at /rustc/c18a5e8a5b1afb0d7a582fe9ebad4c1996c90da3/library/core/src/fmt/mod.rs:1230:17
10: std::io::Write::write_fmt
at /rustc/c18a5e8a5b1afb0d7a582fe9ebad4c1996c90da3/library/std/src/io/mod.rs:1682:15
11: <&std::io::stdio::Stdout as std::io::Write>::write_fmt
at /rustc/c18a5e8a5b1afb0d7a582fe9ebad4c1996c90da3/library/std/src/io/stdio.rs:715:9
12: ::write_fmt
at /rustc/c18a5e8a5b1afb0d7a582fe9ebad4c1996c90da3/library/std/src/io/stdio.rs:689:9
13: std::io::stdio::print_to
at /rustc/c18a5e8a5b1afb0d7a582fe9ebad4c1996c90da3/library/std/src/io/stdio.rs:1007:21
14: std::io::stdio::_print
at /rustc/c18a5e8a5b1afb0d7a582fe9ebad4c1996c90da3/library/std/src/io/stdio.rs:1074:5
15: arrow_cast::pretty::print_batches
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/arrow-cast-38.0.0/src/pretty.rs:62:5
16: datafusion::dataframe::DataFrame::show::{{closure}}
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/datafusion-24.0.0/src/dataframe.rs:682:12
17: query_playground::main::{{closure}}
at ./src/query_playground.rs:785:14
18: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/park.rs:283:63
19: tokio::runtime::coop::with_budget
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/coop.rs:102:5
20: tokio::runtime::coop::budget
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/coop.rs:68:5
21: tokio::runtime::park::CachedParkThread::block_on
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/park.rs:283:31
22: tokio::runtime::context::BlockingRegionGuard::block_on
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/context.rs:315:13
23: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/scheduler/multi_thread/mod.rs:66:9
24: tokio::runtime::runtime::Runtime::block_on
at /Users/dadepo/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/runtime.rs:284:45
25: query_playground::main
at ./src/query_playground.rs:812:5
26: core::ops::function::FnOnce::call_once
at /rustc/c18a5e8a5b1afb0d7a582fe9ebad4c1996c90da3/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
```

### To Reproduce

Create a table with a large number of rows and use array_agg on one of its columns

### Expected behavior

array_agg should not panic on large rows.

### Additional context

```
datafusion = { version = "24.0.0" }
```

Contributor guide

Open the contributing guide

Research direction

Reproduce the large-row array_agg query described in the issue and inspect the reported path through DataFrame::show, arrow_cast::pretty::print_batches, and comfy-table formatting. Use src/query_playground.rs around lines 785 and 812 as the reproduction entry point; done means displaying the result no longer panics on large rows.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.