pingcap / pingcap/tidb

importsdk: support automatic mapping for Aurora/RDS snapshot exports

Open
#70,584 0 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Feature request

TiDB Cloud Premium Import uses `pkg/importsdk` to scan an S3 prefix and build automatic table mappings. Native Amazon Aurora MySQL and Amazon RDS for MySQL snapshot exports use paths such as:

```text
//.//part-....gz.parquet
//./part-....gz.parquet
```

The Parquet reader already supports the files, but the default mydump file router does not recognize this directory layout. A part file can consequently be interpreted from its basename instead of its parent `.` directory, producing incorrect automatic mappings.

## Expected behavior

The Import SDK should recognize Aurora/RDS snapshot-export layouts by default, without a source-layout option or user-provided file route rule.

For an Aurora/RDS layout, automatic scanning should:

- group all physical Parquet part files into one logical mapping per source database and table;
- support current batched and older direct-part layouts;
- use the complete directory hierarchy rather than relying on part filenames or Parquet footer metadata;
- return only a complete, unambiguous mapping set;
- reject mixed, ambiguous, incomplete, unsupported, or multi-export-root input instead of returning partial automatic mappings;
- report the detected layout, table/object counts, physical object bytes, and enough inventory information for the caller to detect changes between Pre-check and job creation;
- preserve existing Dumpling/generic source behavior and custom file-router support.

`IMPORT INTO` syntax and Parquet execution semantics should not change unless end-to-end validation finds a runtime defect.

## Proposed design

1. Add a reusable structural Aurora/RDS path matcher and place its built-in router before the broad generic Parquet rule.
2. Preserve the raw object inventory and scan completeness from the existing single `MDLoader` object listing so `pkg/importsdk` can compare routed mappings with the complete source.
3. Add an Import SDK source-scan result that contains the detected layout, complete table mappings, inventory summary/evidence, and a stable inventory digest. Keep existing `GetTableMetas` compatibility.
4. Return structured scan errors that callers can map to actionable Pre-check messages.
5. Continue to use the existing component-aware wildcard generation and exact-match validation where possible.

## Acceptance criteria

- Correct mappings for current and old AWS layouts, multiple databases/tables/batches, and single-file tables.
- No cross-table wildcard matches.
- Mixed layouts, multiple export roots, unmatched Parquet objects, unsupported identifiers, and truncated scans fail without partial mappings.
- Existing Dumpling/generic/custom-router tests remain unchanged in behavior.
- Scan results expose storage-reported object counts/bytes separately from estimated import size.

## References

- [AWS snapshot-export file naming](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ExportSnapshot.FileNames.html)
- [TiDB: Migrate Data from Amazon Aurora to TiDB](https://docs.pingcap.com/tidb/stable/migrate-from-aurora-using-lightning/)
- [TiDB Cloud data-import naming conventions](https://docs.pingcap.com/tidbcloud/naming-conventions-for-data-import/#aurora-snapshot)

Contributor guide

Open the contributing guide

Research direction

Read pkg/importsdk and trace the existing single MDLoader object listing, default mydump file router, Parquet reader, and GetTableMetas compatibility path. Review the existing router and Import SDK tests first. Done means current and legacy Aurora/RDS layouts produce complete unambiguous mappings, while mixed or incomplete inventories fail without partial results and existing source behavior remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
cloud, data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.