apache / apache/datafusion

Splitting file name after first dot (instead of stripping a '.parquet' suffix) could lead to issues with files containing more than one dot

Open
#17,880 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

Hey 👋

I'm currently facing an issue with [parquet-viewer](https://github.com/XiangpengHao/parquet-viewer) with filenames containing more than one dot. 'parquet-viewer' uses Datafusion underlyingly and the error message I'm seeing definitly comes from Datafusion, so I'm vaguely feeling that potentially the issue could lay around here somewhere.

For reference, [this](https://github.com/XiangpengHao/parquet-viewer/issues/65) is the issue I'm seeing: if a file is named `test.[random-strings].parquet`, it will lead to this error:
```
Plan(
"failed to resolve schema: test",
)
```

The error definitely is raised [here](https://github.com/apache/datafusion/blob/3ee52f85fdb94544da04f6a67f0c7fc03c714843/datafusion/core/src/execution/session_state.rs#L323).

At least when I try to reproduce the issue with Datafusion from Python, I can't seem be able to reproduce the issue though:
```
>>> from datafusion import SessionContext
>>> ctx = SessionContext()
>>> df = ctx.read_parquet("[random-path]/test.ako.parquet")
>>> df.show()
DataFrame()
+----+-----+-----+
| l1 | bar | foo |
+----+-----+-----+
| | | 0 |
| | 0 | |
+----+-----+-----+
>>> df.limit(2)
DataFrame()
+----+-----+-----+
| l1 | bar | foo |
+----+-----+-----+
| | | 0 |
| | 0 | |
+----+-----+-----+
>>> df.schema()
l1: string_view
bar: uint64
foo: uint64
```

I did however find [this](https://github.com/apache/datafusion/blob/3ee52f85fdb94544da04f6a67f0c7fc03c714843/datafusion/catalog/src/listing_schema.rs#L119) line in the Datafusion codebase, which definitely seems fishy to me, as it could lead to problems with multiple parquet files called `part.1.parquet`, `part.2.parquet`?

Maybe it is also connected to the issue I'm seeing here?

Contributor guide

Open the contributing guide

Research direction

Start by reading datafusion/catalog/src/listing_schema.rs around line 119 and datafusion/core/src/execution/session_state.rs around line 323. Reproduce the behavior with multiple-dot parquet filenames and compare it with the Python example. Done means the relevant filename handling is verified or corrected and the reported schema-resolution failure is covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.