apache / apache/datafusion

ListingTable and FileScanConfig assume all files accessible via single ObjectStore instance

Open
#15,964 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

When attempting to read a table of files across multiple S3 buckets in a single query, the query will fail stating that files can't be found:
`Object Store error: Object at location test.parquet not found: Client error with status 404 Not Found: No Body`

This can be seen when the DataFrame is explained.

### To Reproduce

With an AWS account, create two buckets and place a test Parquet file in each.
Create an AmazonS3 ObjectStore instance for each file and register them via SessionContext::register_object_store with the S3 URIs.
Create a DataFusion query with SessionContext::read_parquet() that reads from both files.
Attempting to execute the query or show the explained query will show an error.

### Expected behavior

Query should execute and produce same result as if files were in same S3 bucket.

### Additional context

Amazon S3 ObjectStore instances are specific to the S3 bucket they read from. E.g. S3://bucket1/blah.parquet and S3://bucket2//blah2.parquet require two DIFFERENT ObjectStore instances.

The bug occurs in ListingTable which assumes all files in a table can be accessed via a single ObjectStore. This can be seen in numerous places such as the private list_files_for_scan function and the scan function. ListingTable creates the ObjectStore by just querying for the store associated with the first files in its list.

The same assumption is present in FileScanConfig

Contributor guide

Open the contributing guide

Research direction

Start by tracing ListingTable, especially the private list_files_for_scan and scan functions, then inspect FileScanConfig where the single-ObjectStore assumption is also present. Reproduce the two-bucket S3 query described in the issue, and verify that execution and explanation succeed when files use different registered ObjectStore instances.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.