Feature Request: TiDB Backup to Open Format (Iceberg) Converter

TiDB lacks a scalable, decoupled way to make its data available for OLAP workloads in open formats (Iceberg/Parquet). The existing paths all have fundamental limitations:

- **TiSpark** couples analytical reads to the live OLTP cluster, degrading performance and requiring dedicated read-only TiKV replicas. It is unreliable at scale and does not guarantee compatibility with TiDB v7.0+.
- **TiFlash** requires separate columnar replicas of all data — high infrastructure cost and approaching end-of-life.
- **Dumpling** is single-machine and cannot scale beyond small datasets.

What users actually want is simple: TiDB data queryable in open formats (Iceberg, Parquet) by standard engines (Trino, Spark SQL) — without impacting their OLTP cluster and without vendor lock-in.

**Describe the feature you'd like:**

A tool that converts TiDB data into queryable Apache Iceberg tables, decoupled from the live cluster. The approach: read TiDB's existing BR snapshot backups directly from object storage and convert them to Iceberg.

BR already produces full-cluster snapshots (SST files in S3/GCS) on a regular schedule. These contain complete, consistent point-in-time data. By parsing these files directly — rather than querying a live cluster — we get a scalable, reliable, zero-impact path to open-format analytics.

Combined with TiCDC for incremental updates, this forms a complete real-time TiDB OLAP solution:
- **Backup converter** provides the initial snapshot (bootstrap)
- **TiCDC** provides the continuous stream (incremental changes merged into the same Iceberg table)

Multiple CDC consumption patterns work on top of the base table: a streaming processor like Flink+Kafka (as in [Marlin](https://medium.com/pinterest-engineering/marlin-near-real-time-data-ingestion-for-the-lakehouse-6ea70189e269)) for near-real-time freshness, or TiCDC's S3 sink with periodic Spark MERGE for exact TSO-level consistency.

A full design proposal is available in the accompanying PR: https://github.com/pingcap/tidb/pull/68150

**Describe alternatives you've considered:**

| Alternative | Why not sufficient |
|---|---|
| TiSpark | Coupled to live cluster; degrades OLTP; unreliable at scale; no v7.0+ compat |
| TiFlash | High cost (full columnar replicas); approaching end-of-life |
| Dumpling | Single-machine; cannot scale to PB-level |
| Custom Java/Rust reimplementation | Must reimplement TiDB's SST/MVCC/codec from scratch; high bug risk |

**Teachability, Documentation, Adoption, Migration Strategy:**

**How users would use this:**

```bash
# Convert a table's latest backup snapshot to Iceberg
spark-submit backup-converter.jar \
--backup_parent_dir s3://bucket/backup/cluster/ \
--database mydb \
--table mytable \
--output_table analytics.mydb_mytable \
--iceberg_bucket s3://output-bucket
```

**Production validation:**

At Pinterest, we have used this tool in production to onboard hundreds of tables totaling multiple petabytes of data — replacing TiSpark entirely for snapshot workloads:
- 100% data correctness (row-count + column-level checksum match against TiSpark, verified across all TiDB data types)
- Near-perfect reliability (vs. frequent TiSpark failures requiring retries)
- Compatible with TiDB v8.1 through v8.5 backup formats
- Zero impact on OLTP clusters

**Contribution intent:**

The tool is implemented in Go, directly reusing TiDB's own packages (`tablecodec`, `rowcodec`, `types`, `kvproto`) for correctness. We would like to contribute it upstream so the broader TiDB community can benefit and PingCAP engineers can review the implementation.

**Open question — repository structure:**

The tool has two layers: a Go core library (SST parsing, MVCC, row decoding, Arrow output) and a Scala/Spark application (distributed orchestration, Iceberg write). We'd like guidance from the TiDB community on where these should live:

- **Option A**: Go library in `pingcap/tidb` (e.g., `br/pkg/export/` or `tools/backup-reader/`), plus a standalone CLI that converts backups to CSV/Parquet without Spark. Scala/Spark layer in a separate new repo.
- **Option B**: Everything in a new dedicated repo (e.g., `pingcap/tidb-backup-export`) — Go library, standalone CLI, and Spark integration together.
- **Option C**: Go library in `pingcap/tidb` alongside BR, Scala/Spark layer contributed to an existing analytics-oriented repo (e.g., TiSpark's repo or a new one).

We lean toward contributing the Go core to `pingcap/tidb` (since it imports TiDB internal packages and should stay in sync with format changes) and offering a standalone single-machine CLI as well. The Spark layer could live separately since it has different build tooling (Bazel/SBT) and release cadence. But we're open to whatever structure works best for the community.