[java] cannot read datasets with many columns
- Dominant language
- Java
- Stars
- 94
- Forks
- 152
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 11
Description
### Describe the bug, including details regarding any error messages, version, and platform.
Dataset API gets stuck if one tries to read many columns (100k+) from a parquet or arrow file. The API works if the amount of columns is narrowed via projection.
**Reproduction:**
Use the code from the [example](https://arrow.apache.org/cookbook/java/dataset.html#constructing-datasets) and try to read a file with 100k columns. No batch will be loaded even if the batch size is 1.
version: arrow-dataset 11.0
### Component(s)
Java
Contributor guide
Research direction
Start with the Java Dataset API and the linked dataset construction example, reproducing the hang with a parquet or Arrow file containing 100k+ columns and a batch size of 1. Trace the read path while comparing it with a narrowed projection; done means batches load without getting stuck for the reported wide dataset case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100