apache / apache/incubator-xtable
Parquet source derives the table schema from a single file footer
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 212
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 16
Description
`ParquetConversionSource` builds the table schema from exactly one file's footer. `createInternalTableFromFile` (ParquetConversionSource.java:88) reads a single `LocatedFileStatus`, and every caller feeds it one file selected by `getMostRecentParquetFile` — `getCurrentSnapshot` via `getMostRecentTable` (:206), and the table-change path at :190-191.
Two consequences for a Parquet directory treated as a table:
1. **Schema evolution loses columns.** If a column was added (or dropped) partway through the table's history, the reported schema is whichever generation the newest-by-modification-time file happens to belong to. Files from other generations are then listed as data files whose columns do not all exist in the read schema (or which lack columns the schema promises). Modification time is also not a stable proxy for "newest schema": a backup/restore, `distcp`, or object-store copy rewrites mtimes, so the same directory can report different schemas before and after a copy.
2. **A Hive-partitioned directory can report a partition field with no backing column.** `partitionSpecExtractor.spec(schema)` resolves the configured partition paths against that single footer's schema. When the partition column exists only in the directory structure (`region=eu/...`) and not inside the files — common for Hive-written data — the partition field resolves against nothing, while the values remain derivable from the paths.
A possible direction for (1): merge the footers of all listed files into the read schema (fields present in any file, with a hard error when two files disagree on a field's type), rather than trusting one file. For (2): synthesize the partition column into the read schema, typed from the observed path values, when the configured partition path is absent from the footers.
Happy to provide a small fixture directory (two file generations plus a Hive-partitioned layout) if useful.
*This issue was created with AI assistance.*
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in ParquetConversionSource.java at createInternalTableFromFile (line 88), then trace getMostRecentTable and getCurrentSnapshot (line 206) plus the table-change path (lines 190-191). Check how getMostRecentParquetFile supplies the footer and how partitionSpecExtractor.spec(schema) resolves paths. Done means the schema accounts for all listed file footers, rejects conflicting field types, and handles configured partition fields absent from file schemas.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100