Splitting file name after first dot (instead of stripping a '.parquet' suffix) could lead to issues with files containing more than one dot
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
Hey 👋
I'm currently facing an issue with [parquet-viewer](https://github.com/XiangpengHao/parquet-viewer) with filenames containing more than one dot. 'parquet-viewer' uses Datafusion underlyingly and the error message I'm seeing definitly comes from Datafusion, so I'm vaguely feeling that potentially the issue could lay around here somewhere.
For reference, [this](https://github.com/XiangpengHao/parquet-viewer/issues/65) is the issue I'm seeing: if a file is named `test.[random-strings].parquet`, it will lead to this error:
```
Plan(
"failed to resolve schema: test",
)
```
The error definitely is raised [here](https://github.com/apache/datafusion/blob/3ee52f85fdb94544da04f6a67f0c7fc03c714843/datafusion/core/src/execution/session_state.rs#L323).
At least when I try to reproduce the issue with Datafusion from Python, I can't seem be able to reproduce the issue though:
```
>>> from datafusion import SessionContext
>>> ctx = SessionContext()
>>> df = ctx.read_parquet("[random-path]/test.ako.parquet")
>>> df.show()
DataFrame()
+----+-----+-----+
| l1 | bar | foo |
+----+-----+-----+
| | | 0 |
| | 0 | |
+----+-----+-----+
>>> df.limit(2)
DataFrame()
+----+-----+-----+
| l1 | bar | foo |
+----+-----+-----+
| | | 0 |
| | 0 | |
+----+-----+-----+
>>> df.schema()
l1: string_view
bar: uint64
foo: uint64
```
I did however find [this](https://github.com/apache/datafusion/blob/3ee52f85fdb94544da04f6a67f0c7fc03c714843/datafusion/catalog/src/listing_schema.rs#L119) line in the Datafusion codebase, which definitely seems fishy to me, as it could lead to problems with multiple parquet files called `part.1.parquet`, `part.2.parquet`?
Maybe it is also connected to the issue I'm seeing here?
Contributor guide
Research direction
Start by reading datafusion/catalog/src/listing_schema.rs around line 119 and datafusion/core/src/execution/session_state.rs around line 323. Reproduce the behavior with multiple-dot parquet filenames and compare it with the Python example. Done means the relevant filename handling is verified or corrected and the reported schema-resolution failure is covered by a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100