Cannot register table to represent multiple parquet files in S3 bucket
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
**Describe the bug**
This works:
```sql
CREATE EXTERNAL TABLE yellow_2019_01 STORED AS PARQUET LOCATION "s3://ossb-nyctaxi/yellow/2019/yellow_tripdata_2019-01.parquet";
```
This does not work:
```sql
CREATE EXTERNAL TABLE yellow_2019 STORED AS PARQUET LOCATION "s3://ossb-nyctaxi/yellow/2019";
```
Fails with:
```
ObjectStore(NotFound { path: "yellow/2019", source: Error { retries: 0, message: "No Body", source: Some(reqwest::Error { kind: Status(404), url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("s3.us-east-2.amazonaws.com")), port: None, path: "/ossb-nyctaxi/yellow/2019", query: None, fragment: None } }) } }
```
**To Reproduce**
As described.
**Expected behavior**
Should be able to register a directory containing parquet files.
**Additional context**
Possibly related to https://github.com/apache/arrow-datafusion/issues/1736
Contributor guide
Research direction
Start with CREATE EXTERNAL TABLE handling for S3 locations and reproduce the two SQL statements from the issue. Trace how a directory path is resolved versus a single parquet object; done means a directory containing parquet files can be registered as one external table.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, rust
- Domain
- cloud, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 30/100