apache / apache/incubator-xtable

Parquet source derives the table schema from a single file footer

Open
#901 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
1.2k
Forks
212
Avg merge
4d 9h
Merged PRs (30d)
16

Description

`ParquetConversionSource` builds the table schema from exactly one file's footer. `createInternalTableFromFile` (ParquetConversionSource.java:88) reads a single `LocatedFileStatus`, and every caller feeds it one file selected by `getMostRecentParquetFile` — `getCurrentSnapshot` via `getMostRecentTable` (:206), and the table-change path at :190-191.

Two consequences for a Parquet directory treated as a table:

1. **Schema evolution loses columns.** If a column was added (or dropped) partway through the table's history, the reported schema is whichever generation the newest-by-modification-time file happens to belong to. Files from other generations are then listed as data files whose columns do not all exist in the read schema (or which lack columns the schema promises). Modification time is also not a stable proxy for "newest schema": a backup/restore, `distcp`, or object-store copy rewrites mtimes, so the same directory can report different schemas before and after a copy.

2. **A Hive-partitioned directory can report a partition field with no backing column.** `partitionSpecExtractor.spec(schema)` resolves the configured partition paths against that single footer's schema. When the partition column exists only in the directory structure (`region=eu/...`) and not inside the files — common for Hive-written data — the partition field resolves against nothing, while the values remain derivable from the paths.

A possible direction for (1): merge the footers of all listed files into the read schema (fields present in any file, with a hard error when two files disagree on a field's type), rather than trusting one file. For (2): synthesize the partition column into the read schema, typed from the observed path values, when the configured partition path is absent from the footers.

Happy to provide a small fixture directory (two file generations plus a Hive-partitioned layout) if useful.

*This issue was created with AI assistance.*

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in ParquetConversionSource.java at createInternalTableFromFile (line 88), then trace getMostRecentTable and getCurrentSnapshot (line 206) plus the table-change path (lines 190-191). Check how getMostRecentParquetFile supplies the footer and how partitionSpecExtractor.spec(schema) resolves paths. Done means the schema accounts for all listed file footers, rejects conflicting field types, and handles configured partition fields absent from file schemas.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.