apache / apache/parquet-java

Add integration tests for all `parquet-testing` files

Aperta
#3,670 4 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Java
Stelle
3.1k
Fork
1.6k
Merge medio
3g 12h
PR unite (30g)
33

Descrizione

## Feature request

Add integration tests that run against every Parquet file in the [`apache/parquet-testing`](https://github.com/apache/parquet-testing) repository.

## Problem

`parquet-java` currently [tests only selected files](https://github.com/apache/parquet-java/blob/b6fb9e2f4d08961d32d88ff6e45f454d16c32255/parquet-hadoop/src/test/java/org/apache/parquet/hadoop/InterOpTester.java#L36) from `parquet-testing`. As a result, new interoperability fixtures are not automatically tested, and compatibility regressions may go undetected.

There are already several [`parquet-java` issues involving files from `parquet-testing`](https://github.com/apache/parquet-java/issues?q=is%3Aissue%20state%3Aopen%20parquet%20testing).

## Proposed solution

Add integration tests that classify files into two groups:

- **Valid files:** must open successfully and allow all row groups and columns to be read.
- **Invalid files:** must fail during footer parsing or record decoding with an expected exception.

`InterOpTester` already supports downloading individual files from `parquet-testing`. This feature would extend that approach to provide repository-wide coverage.

## Acceptance criteria

- [ ] Every Parquet file in the pinned revision is classified.
- [ ] All valid files are read successfully.
- [ ] All invalid files fail as expected.
- [ ] Each file is reported as an independent test.
- [ ] Unclassified files cause the test to fail.
- [ ] Exclusions require an explicit entry linked to a tracking issue.

## Related issues

- [#3336: Hadoop parquet reader and parquet CLI fail on some files from `parquet-testing`](https://github.com/apache/parquet-java/issues/3336)
- [#1686: Add integration test for BYTE_STREAM_SPLIT](https://github.com/apache/parquet-java/issues/1686)
- [#3633: Older `parquet-java` readers fail on files with unprojected VARIANT columns](https://github.com/apache/parquet-java/issues/3633)

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Inizia da parquet-hadoop/src/test/java/org/apache/parquet/hadoop/InterOpTester.java e verifica come scarica e testa i file parquet-testing selezionati. Esegui i test di interoperabilità esistenti, quindi assegna ogni file nella revisione fissata di parquet-testing a un caso valido o non valido. Il lavoro è completato quando tutti i file vengono segnalati in modo indipendente, i file validi vengono letti correttamente, i file non validi falliscono come previsto e i file non classificati fanno fallire il test.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
java
Ambito
testing-qa
Tipo di issue
Funzionalità
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Tranquilla
Chiarezza
Abbastanza chiara
Idoneità per principianti
55/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.