apache / apache/datafusion

Run tpch q5 with 100G data got error

Open
#3,291 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Describe the bug**
```
Error: ArrowError(ExternalError(ArrowError(ExternalError(Execution("Arrow error: External error: Arrow error: External error: Execution error: Arrow error: External error: Arrow error: External error: Execution error: Arrow error: External error: Arrow error: External error: Execution error: Arrow error: External error: Arrow error: External error: Execution error: Arrow error: External error: Arrow error: External error: Execution error: Arrow error: External error: Execution error: Arrow error: Parser error: Error parsing line 14999998: Error(UnequalLengths { pos: Some(Position { byte: 2463490006, line: 14999999, record: 14999998 }), expected_len: 9, len: 3 })")))))
```

**To Reproduce**
command:
```
./tpch-gen.sh 100
cargo run --release --bin tpch -- benchmark datafusion --iterations 1 --path ./data --format tbl --query 5 --batch-size 4096
```

**Expected behavior**
A clear and concise description of what you expected to happen.

**Additional context**
Add any other context about the problem here.

Contributor guide

Open the contributing guide

Research direction

Start by running tpch-gen.sh 100 and the provided cargo run --release --bin tpch command to reproduce the parser error. Inspect the generated data around line 14999999 and the tpch benchmark entry point to determine why the record has three fields instead of nine. Done means query 5 completes successfully on the 100G dataset.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.