IMPORT INTO silently shifts Parquet LIST elements across rows and loses data
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
# [IMPORT INTO] Parquet LIST values can be silently shifted across rows and lost
## Bug Report
### 1. Minimal reproduce step (Required)
The following real-TiKV test uses Apache Arrow's high-level writer to create a standard
three-level Parquet `LIST` column. The four source rows are:
```text
1 [101,102] row-1
2 [201] row-2
3 [] row-3
4 [401,402,403] row-4
```
Add this test to `tests/realtikvtest/importintotest` on current master:
```go
package importintotest
import (
"fmt"
"os"
"path/filepath"
"github.com/apache/arrow-go/v18/arrow"
"github.com/apache/arrow-go/v18/arrow/array"
"github.com/apache/arrow-go/v18/arrow/memory"
"github.com/apache/arrow-go/v18/parquet"
"github.com/apache/arrow-go/v18/parquet/pqarrow"
"github.com/pingcap/tidb/pkg/testkit"
)
func (s *mockGCSSuite) writeParquetList(fileName string) {
f, err := os.Create(fileName)
s.Require().NoError(err)
arrowSchema := arrow.NewSchema([]arrow.Field{
{Name: "id", Type: arrow.PrimitiveTypes.Int64},
{Name: "items", Type: arrow.ListOf(arrow.PrimitiveTypes.Int64), Nullable: true},
{Name: "marker", Type: arrow.BinaryTypes.String},
}, nil)
w, err := pqarrow.NewFileWriter(
arrowSchema,
f,
parquet.NewWriterProperties(),
pqarrow.DefaultWriterProps(),
)
s.Require().NoError(err)
b := array.NewRecordBuilder(memory.DefaultAllocator, arrowSchema)
b.Field(0).(*array.Int64Builder).AppendValues([]int64{1, 2, 3, 4}, nil)
b.Field(2).(*array.StringBuilder).AppendValues(
[]string{"row-1", "row-2", "row-3", "row-4"}, nil,
)
lists := b.Field(1).(*array.ListBuilder)
values := lists.ValueBuilder().(*array.Int64Builder)
lists.Append(true)
values.AppendValues([]int64{101, 102}, nil)
lists.Append(true)
values.Append(201)
lists.Append(true) // empty list
lists.Append(true)
values.AppendValues([]int64{401, 402, 403}, nil)
record := b.NewRecordBatch()
s.Require().NoError(w.Write(record))
record.Release()
b.Release()
s.Require().NoError(w.Close())
}
func (s *mockGCSSuite) TestParquetListPreservesTopLevelRows() {
fileName := filepath.Join(s.T().TempDir(), "list.parquet")
s.writeParquetList(fileName)
s.tk.MustExec("DROP TABLE IF EXISTS test.parquet_list")
s.tk.MustExec(`CREATE TABLE test.parquet_list(
id BIGINT PRIMARY KEY,
items JSON,
marker VARCHAR(32)
)`)
err := s.tk.QueryToErr(fmt.Sprintf(
"IMPORT INTO test.parquet_list FROM '%s' FORMAT 'parquet'", fileName,
))
if err != nil {
// LIST is explicitly unsupported today, so a rejection before any write is valid.
s.Require().ErrorContains(err, "unsupported parquet")
s.tk.MustQuery("SELECT COUNT(*) FROM test.parquet_list").Check(testkit.Rows("0"))
return
}
// If LIST is accepted, every top-level row boundary must be preserved.
s.tk.MustExec("ADMIN CHECK TABLE test.parquet_list")
s.tk.MustQuery(
"SELECT id, CAST(items AS CHAR), marker FROM test.parquet_list ORDER BY id",
).Check(testkit.Rows(
`1 [101, 102] row-1`,
`2 [201] row-2`,
`3 [] row-3`,
`4 [401, 402, 403] row-4`,
))
}
```
Run it with a real TiKV cluster:
```bash
./tests/realtikvtest/scripts/classic/bootstrap-test-with-cluster.sh \
go test ./tests/realtikvtest/importintotest -v --tags=intest -with-real-tikv \
-run 'TestImportInto/TestParquetListPreservesTopLevelRows' -timeout 10m
```
### 2. What did you expect to see? (Required)
The current reader declares Parquet `LIST` unsupported, so `IMPORT INTO` should reject the file
before writing any row. If LIST support is enabled, it must preserve the four arrays and their
top-level row boundaries shown above.
### 3. What did you see instead (Required)
`IMPORT INTO` reports `finished`, imports four rows, and `ADMIN CHECK TABLE` passes. The persisted
rows are nevertheless:
```text
1 101 row-1
2 102 row-2
3 201 row-3
4 NULL row-4
```
Element `102` belongs to source row 1 but is attached to row 2. Element `201` belongs to source
row 2 but is attached to row 3. All three elements from source row 4 are lost.
This reproduced twice on fresh tables in a Premium Next Generation deployment using an
S3-compatible source, distributed import workers, default checksum, and metadata locking enabled.
On the same environment, a sibling file that changes only `items` from `LIST` to scalar
`INT64` imports the expected values `101, 201, 301, 401`.
The reader already lists `schema.ConvertedTypes.List` in `unsupportedParquetTypes`, but schema
validation checks the primitive leaf descriptor only. In a standard three-level encoding, the
`LIST` annotation is on an ancestor group while the leaf is plain `INT64`, so validation misses it.
The generic `columnIterator.Next` then consumes one definition-level entry per SQL row and never
uses the buffered repetition levels. `rowGroupParser` stops after the Parquet top-level row count.
Repeated elements are therefore consumed as later rows, while entries remaining after the
top-level row count are discarded.
A counterfactual that only walks each leaf's ancestor groups and rejects unsupported annotations
makes the focused parser test and the full real-TiKV import test pass: the LIST import fails before
writes, the target remains empty, and the scalar sibling still imports correctly.
#### Relation to #67856 and #65487
Issue #67856 and PR #65487 propose adding `LIST` to `VECTOR` support in the older Lightning
reader. This report does not require that enhancement. It covers the current
`pkg/dumpformat/parquetfile` path accepting an unsupported standard `LIST` file and
silently changing data. Rejecting the parent LIST annotation before writes is sufficient to fix
the current corruption; a future implementation must additionally group entries by repetition
level, as #65487's dedicated list iterator does.
#### Production impact
This can occur in an ordinary data-lake workflow: Arrow, PyArrow, Spark, Hive, and similar tools
write an array/list column using Parquet's standard LIST encoding, and the file is imported into a
JSON-compatible target column. One normal import is enough; no malformed file, fault, concurrency,
retry, restart, permissive SQL mode, partitioning, or disabled MDL is required.
- Mechanism event: one import of a standard Parquet file containing a repeated LIST column.
- Immediate and permanent symptom: elements move to different logical rows and later elements are
omitted while the job reports success.
- Self-heal path: none; later reads return the changed rows indefinitely.
- Irreversible consumer: the normal IMPORT INTO data-writing path.
- Independent event count: one.
Default checksum and `ADMIN CHECK TABLE` operate on KVs produced after the wrong row assembly, so
they certify internal consistency without comparing the persisted rows with the source semantics.
### 4. What is your TiDB version? (Required)
```text
Release Version: CLOUD.202608.0-16c97eb67f
Edition: Enterprise
Git Commit Hash: 16c97eb67f9558f39535bf63b8c2ffe388fbe391
Git Branch: HEAD
Store: tikv
Kernel Type: Next Generation
```
The same root was also reproduced locally at that exact TiDB commit with one TiDB, one PD, and one
real TiKV.
Contributor guide
Research direction
Start in pkg/dumpformat/parquetfile, where Parquet schema validation and row parsing are described, and inspect the focused test requested under tests/realtikvtest/importintotest. Run the provided real-TiKV test command to reproduce the shifted LIST rows. Done means the unsupported LIST file is rejected before writes with an empty target, while the scalar sibling still imports correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100