importsdk: support automatic mapping for Aurora/RDS snapshot exports
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Feature request
TiDB Cloud Premium Import uses `pkg/importsdk` to scan an S3 prefix and build automatic table mappings. Native Amazon Aurora MySQL and Amazon RDS for MySQL snapshot exports use paths such as:
```text
//.//part-....gz.parquet
//./part-....gz.parquet
```
The Parquet reader already supports the files, but the default mydump file router does not recognize this directory layout. A part file can consequently be interpreted from its basename instead of its parent `.` directory, producing incorrect automatic mappings.
## Expected behavior
The Import SDK should recognize Aurora/RDS snapshot-export layouts by default, without a source-layout option or user-provided file route rule.
For an Aurora/RDS layout, automatic scanning should:
- group all physical Parquet part files into one logical mapping per source database and table;
- support current batched and older direct-part layouts;
- use the complete directory hierarchy rather than relying on part filenames or Parquet footer metadata;
- return only a complete, unambiguous mapping set;
- reject mixed, ambiguous, incomplete, unsupported, or multi-export-root input instead of returning partial automatic mappings;
- report the detected layout, table/object counts, physical object bytes, and enough inventory information for the caller to detect changes between Pre-check and job creation;
- preserve existing Dumpling/generic source behavior and custom file-router support.
`IMPORT INTO` syntax and Parquet execution semantics should not change unless end-to-end validation finds a runtime defect.
## Proposed design
1. Add a reusable structural Aurora/RDS path matcher and place its built-in router before the broad generic Parquet rule.
2. Preserve the raw object inventory and scan completeness from the existing single `MDLoader` object listing so `pkg/importsdk` can compare routed mappings with the complete source.
3. Add an Import SDK source-scan result that contains the detected layout, complete table mappings, inventory summary/evidence, and a stable inventory digest. Keep existing `GetTableMetas` compatibility.
4. Return structured scan errors that callers can map to actionable Pre-check messages.
5. Continue to use the existing component-aware wildcard generation and exact-match validation where possible.
## Acceptance criteria
- Correct mappings for current and old AWS layouts, multiple databases/tables/batches, and single-file tables.
- No cross-table wildcard matches.
- Mixed layouts, multiple export roots, unmatched Parquet objects, unsupported identifiers, and truncated scans fail without partial mappings.
- Existing Dumpling/generic/custom-router tests remain unchanged in behavior.
- Scan results expose storage-reported object counts/bytes separately from estimated import size.
## References
- [AWS snapshot-export file naming](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ExportSnapshot.FileNames.html)
- [TiDB: Migrate Data from Amazon Aurora to TiDB](https://docs.pingcap.com/tidb/stable/migrate-from-aurora-using-lightning/)
- [TiDB Cloud data-import naming conventions](https://docs.pingcap.com/tidbcloud/naming-conventions-for-data-import/#aurora-snapshot)
Contributor guide
Research direction
Read pkg/importsdk and trace the existing single MDLoader object listing, default mydump file router, Parquet reader, and GetTableMetas compatibility path. Review the existing router and Import SDK tests first. Done means current and legacy Aurora/RDS layouts produce complete unambiguous mappings, while mixed or incomplete inventories fail without partial results and existing source behavior remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- cloud, data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100