pingcap / pingcap/tidb

IMPORT INTO silently shifts Parquet LIST elements across rows and loses data

Open
#70,363 3 comments 0 reactions 0 assignees View on GitHub
component/import found-by-ai type/new-feature
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

# [IMPORT INTO] Parquet LIST values can be silently shifted across rows and lost

## Bug Report

### 1. Minimal reproduce step (Required)

The following real-TiKV test uses Apache Arrow's high-level writer to create a standard
three-level Parquet `LIST` column. The four source rows are:

```text
1 [101,102] row-1
2 [201] row-2
3 [] row-3
4 [401,402,403] row-4
```

Add this test to `tests/realtikvtest/importintotest` on current master:

```go
package importintotest

import (
"fmt"
"os"
"path/filepath"

"github.com/apache/arrow-go/v18/arrow"
"github.com/apache/arrow-go/v18/arrow/array"
"github.com/apache/arrow-go/v18/arrow/memory"
"github.com/apache/arrow-go/v18/parquet"
"github.com/apache/arrow-go/v18/parquet/pqarrow"
"github.com/pingcap/tidb/pkg/testkit"
)

func (s *mockGCSSuite) writeParquetList(fileName string) {
f, err := os.Create(fileName)
s.Require().NoError(err)

arrowSchema := arrow.NewSchema([]arrow.Field{
{Name: "id", Type: arrow.PrimitiveTypes.Int64},
{Name: "items", Type: arrow.ListOf(arrow.PrimitiveTypes.Int64), Nullable: true},
{Name: "marker", Type: arrow.BinaryTypes.String},
}, nil)
w, err := pqarrow.NewFileWriter(
arrowSchema,
f,
parquet.NewWriterProperties(),
pqarrow.DefaultWriterProps(),
)
s.Require().NoError(err)

b := array.NewRecordBuilder(memory.DefaultAllocator, arrowSchema)
b.Field(0).(*array.Int64Builder).AppendValues([]int64{1, 2, 3, 4}, nil)
b.Field(2).(*array.StringBuilder).AppendValues(
[]string{"row-1", "row-2", "row-3", "row-4"}, nil,
)
lists := b.Field(1).(*array.ListBuilder)
values := lists.ValueBuilder().(*array.Int64Builder)
lists.Append(true)
values.AppendValues([]int64{101, 102}, nil)
lists.Append(true)
values.Append(201)
lists.Append(true) // empty list
lists.Append(true)
values.AppendValues([]int64{401, 402, 403}, nil)

record := b.NewRecordBatch()
s.Require().NoError(w.Write(record))
record.Release()
b.Release()
s.Require().NoError(w.Close())
}

func (s *mockGCSSuite) TestParquetListPreservesTopLevelRows() {
fileName := filepath.Join(s.T().TempDir(), "list.parquet")
s.writeParquetList(fileName)

s.tk.MustExec("DROP TABLE IF EXISTS test.parquet_list")
s.tk.MustExec(`CREATE TABLE test.parquet_list(
id BIGINT PRIMARY KEY,
items JSON,
marker VARCHAR(32)
)`)

err := s.tk.QueryToErr(fmt.Sprintf(
"IMPORT INTO test.parquet_list FROM '%s' FORMAT 'parquet'", fileName,
))
if err != nil {
// LIST is explicitly unsupported today, so a rejection before any write is valid.
s.Require().ErrorContains(err, "unsupported parquet")
s.tk.MustQuery("SELECT COUNT(*) FROM test.parquet_list").Check(testkit.Rows("0"))
return
}

// If LIST is accepted, every top-level row boundary must be preserved.
s.tk.MustExec("ADMIN CHECK TABLE test.parquet_list")
s.tk.MustQuery(
"SELECT id, CAST(items AS CHAR), marker FROM test.parquet_list ORDER BY id",
).Check(testkit.Rows(
`1 [101, 102] row-1`,
`2 [201] row-2`,
`3 [] row-3`,
`4 [401, 402, 403] row-4`,
))
}
```

Run it with a real TiKV cluster:

```bash
./tests/realtikvtest/scripts/classic/bootstrap-test-with-cluster.sh \
go test ./tests/realtikvtest/importintotest -v --tags=intest -with-real-tikv \
-run 'TestImportInto/TestParquetListPreservesTopLevelRows' -timeout 10m
```

### 2. What did you expect to see? (Required)

The current reader declares Parquet `LIST` unsupported, so `IMPORT INTO` should reject the file
before writing any row. If LIST support is enabled, it must preserve the four arrays and their
top-level row boundaries shown above.

### 3. What did you see instead (Required)

`IMPORT INTO` reports `finished`, imports four rows, and `ADMIN CHECK TABLE` passes. The persisted
rows are nevertheless:

```text
1 101 row-1
2 102 row-2
3 201 row-3
4 NULL row-4
```

Element `102` belongs to source row 1 but is attached to row 2. Element `201` belongs to source
row 2 but is attached to row 3. All three elements from source row 4 are lost.

This reproduced twice on fresh tables in a Premium Next Generation deployment using an
S3-compatible source, distributed import workers, default checksum, and metadata locking enabled.
On the same environment, a sibling file that changes only `items` from `LIST` to scalar
`INT64` imports the expected values `101, 201, 301, 401`.

The reader already lists `schema.ConvertedTypes.List` in `unsupportedParquetTypes`, but schema
validation checks the primitive leaf descriptor only. In a standard three-level encoding, the
`LIST` annotation is on an ancestor group while the leaf is plain `INT64`, so validation misses it.

The generic `columnIterator.Next` then consumes one definition-level entry per SQL row and never
uses the buffered repetition levels. `rowGroupParser` stops after the Parquet top-level row count.
Repeated elements are therefore consumed as later rows, while entries remaining after the
top-level row count are discarded.

A counterfactual that only walks each leaf's ancestor groups and rejects unsupported annotations
makes the focused parser test and the full real-TiKV import test pass: the LIST import fails before
writes, the target remains empty, and the scalar sibling still imports correctly.

#### Relation to #67856 and #65487

Issue #67856 and PR #65487 propose adding `LIST` to `VECTOR` support in the older Lightning
reader. This report does not require that enhancement. It covers the current
`pkg/dumpformat/parquetfile` path accepting an unsupported standard `LIST` file and
silently changing data. Rejecting the parent LIST annotation before writes is sufficient to fix
the current corruption; a future implementation must additionally group entries by repetition
level, as #65487's dedicated list iterator does.

#### Production impact

This can occur in an ordinary data-lake workflow: Arrow, PyArrow, Spark, Hive, and similar tools
write an array/list column using Parquet's standard LIST encoding, and the file is imported into a
JSON-compatible target column. One normal import is enough; no malformed file, fault, concurrency,
retry, restart, permissive SQL mode, partitioning, or disabled MDL is required.

- Mechanism event: one import of a standard Parquet file containing a repeated LIST column.
- Immediate and permanent symptom: elements move to different logical rows and later elements are
omitted while the job reports success.
- Self-heal path: none; later reads return the changed rows indefinitely.
- Irreversible consumer: the normal IMPORT INTO data-writing path.
- Independent event count: one.

Default checksum and `ADMIN CHECK TABLE` operate on KVs produced after the wrong row assembly, so
they certify internal consistency without comparing the persisted rows with the source semantics.

### 4. What is your TiDB version? (Required)

```text
Release Version: CLOUD.202608.0-16c97eb67f
Edition: Enterprise
Git Commit Hash: 16c97eb67f9558f39535bf63b8c2ffe388fbe391
Git Branch: HEAD
Store: tikv
Kernel Type: Next Generation
```

The same root was also reproduced locally at that exact TiDB commit with one TiDB, one PD, and one
real TiKV.

Contributor guide

Open the contributing guide

Research direction

Start in pkg/dumpformat/parquetfile, where Parquet schema validation and row parsing are described, and inspect the focused test requested under tests/realtikvtest/importintotest. Run the provided real-TiKV test command to reproduce the shifted LIST rows. Done means the unsupported LIST file is rejected before writes with an empty target, while the scalar sibling still imports correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.