Splitting file name after first dot (instead of stripping a '.parquet' suffix) could lead to issues with files containing more than one dot
- Lingua principale
- Rust
- Stelle
- 9.3k
- Fork
- 2.4k
- Merge medio
- 3g 11h
- PR unite (30g)
- 360
Descrizione
Hey 👋
I'm currently facing an issue with [parquet-viewer](https://github.com/XiangpengHao/parquet-viewer) with filenames containing more than one dot. 'parquet-viewer' uses Datafusion underlyingly and the error message I'm seeing definitly comes from Datafusion, so I'm vaguely feeling that potentially the issue could lay around here somewhere.
For reference, [this](https://github.com/XiangpengHao/parquet-viewer/issues/65) is the issue I'm seeing: if a file is named `test.[random-strings].parquet`, it will lead to this error:
```
Plan(
"failed to resolve schema: test",
)
```
The error definitely is raised [here](https://github.com/apache/datafusion/blob/3ee52f85fdb94544da04f6a67f0c7fc03c714843/datafusion/core/src/execution/session_state.rs#L323).
At least when I try to reproduce the issue with Datafusion from Python, I can't seem be able to reproduce the issue though:
```
>>> from datafusion import SessionContext
>>> ctx = SessionContext()
>>> df = ctx.read_parquet("[random-path]/test.ako.parquet")
>>> df.show()
DataFrame()
+----+-----+-----+
| l1 | bar | foo |
+----+-----+-----+
| | | 0 |
| | 0 | |
+----+-----+-----+
>>> df.limit(2)
DataFrame()
+----+-----+-----+
| l1 | bar | foo |
+----+-----+-----+
| | | 0 |
| | 0 | |
+----+-----+-----+
>>> df.schema()
l1: string_view
bar: uint64
foo: uint64
```
I did however find [this](https://github.com/apache/datafusion/blob/3ee52f85fdb94544da04f6a67f0c7fc03c714843/datafusion/catalog/src/listing_schema.rs#L119) line in the Datafusion codebase, which definitely seems fishy to me, as it could lead to problems with multiple parquet files called `part.1.parquet`, `part.2.parquet`?
Maybe it is also connected to the issue I'm seeing here?
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Inizia leggendo datafusion/catalog/src/listing_schema.rs intorno alla riga 119 e datafusion/core/src/execution/session_state.rs intorno alla riga 323. Riproduci il comportamento con nomi di file parquet contenenti più punti e confrontalo con l’esempio Python. Il lavoro è completato quando la gestione rilevante dei nomi di file è stata verificata o corretta e il problema segnalato nella risoluzione dello schema è coperto da un test di regressione.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- python, rust
- Ambito
- data-engineering, databases
- Tipo di issue
- Bug
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Ferma
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 35/100