Automatically call ListingTableConfig.infer()
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
(This section helps Arrow developers understand the context and *why* for this feature, in addition to the *what*)
```rs
let ctx = SessionContext::new();
let s3 = object_store::aws::AmazonS3Builder::from_env()
.with_bucket_name("my-bucket")
.with_region("us-east-1")
.build()?;
ctx.runtime_env()
.register_object_store("s3", "my-bucket", Arc::new(s3));
let uris = vec![
"s3://a/b/c/1.parquet",
"s3://a/b/c/2.parquet",
];
let uris = uris.into_iter().map(|uri| ListingTableUrl::parse(uri).unwrap()).collect::>();
let table = Arc::new(ListingTable::try_new(ListingTableConfig::new_with_multi_paths(uris))?);
let df = ctx.read_table(table)?;
```
gives
```
error: Internal error: No schema provided.. This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker
```
I had to manually call `infer()`:
```rs
let mut config = ListingTableConfig::new_with_multi_paths(uris);
config = config.infer(&ctx.state()).await?;
```
**Describe the solution you'd like**
DataFusion should call `infer()` if config is not set.
Parquet files have schema in them, and it is common to omit schema when reading or writing parquet files.
Contributor guide
Research direction
Start by reproducing the example through ListingTableConfig::new_with_multi_paths, ListingTable::try_new, and ctx.read_table, then inspect how ListingTableConfig::infer(&ctx.state()) is currently used. Done means a missing schema is inferred automatically for the shown Parquet paths and the example succeeds without the manual infer call.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100