apache / apache/datafusion

Automatically call ListingTableConfig.infer()

Open
#5,112 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
(This section helps Arrow developers understand the context and *why* for this feature, in addition to the *what*)

```rs
let ctx = SessionContext::new();
let s3 = object_store::aws::AmazonS3Builder::from_env()
.with_bucket_name("my-bucket")
.with_region("us-east-1")
.build()?;
ctx.runtime_env()
.register_object_store("s3", "my-bucket", Arc::new(s3));
let uris = vec![
"s3://a/b/c/1.parquet",
"s3://a/b/c/2.parquet",
];
let uris = uris.into_iter().map(|uri| ListingTableUrl::parse(uri).unwrap()).collect::>();
let table = Arc::new(ListingTable::try_new(ListingTableConfig::new_with_multi_paths(uris))?);
let df = ctx.read_table(table)?;
```

gives

```
error: Internal error: No schema provided.. This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker
```

I had to manually call `infer()`:

```rs
let mut config = ListingTableConfig::new_with_multi_paths(uris);
config = config.infer(&ctx.state()).await?;
```

**Describe the solution you'd like**

DataFusion should call `infer()` if config is not set.
Parquet files have schema in them, and it is common to omit schema when reading or writing parquet files.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the example through ListingTableConfig::new_with_multi_paths, ListingTable::try_new, and ctx.read_table, then inspect how ListingTableConfig::infer(&ctx.state()) is currently used. Done means a missing schema is inferred automatically for the shown Parquet paths and the example succeeds without the manual infer call.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.