pingcap / pingcap/tidb

lightning, importinto: confusing error when Parquet parser is used on non-Parquet files

Open
#66,948 0 comments 0 reactions 0 assignees View on GitHub
component/lightning type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

### What did you do?

Configure `IMPORT INTO` or Lightning to import a CSV/SQL file but mistakenly set the data format to `parquet`.

### What did you expect to see?

A clear error message indicating the file is not in Parquet format, such as:
```
not a valid Parquet file: missing trailing PAR1 magic
```

### What did you see instead?

A confusing low-level error from the Arrow library about "invalid offset", which comes from the footer parser trying to interpret random bytes as Parquet metadata offsets. This makes it very hard for users to diagnose the misconfiguration.

### Root Cause

The Arrow library's `parseMetaData()` reads the file footer and interprets bytes as offsets/lengths without first validating the Parquet magic number. When a non-Parquet file is passed in, the garbage bytes produce meaningless "invalid offset" errors.

### Proposed Fix

1. Parse the Parquet footer ourselves before handing off to Arrow, with clear error categorization:
- `ErrNotParquet`: file is not Parquet (wrong magic, too small)
- `ErrParquetCorrupt`: file has Parquet magic but broken metadata
- `ErrParquetEncrypted`: encrypted Parquet (PARE magic) is not supported
2. Pass the parsed metadata to Arrow via `file.WithMetadata()` to skip its internal `parseMetaData()`, eliminating double-parse overhead.
3. Use `SourceFileMeta.FileSize` instead of `SeekEnd` to avoid an extra I/O round-trip on cloud storage.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.