Support separate metadata/data location prefixes in RewriteTablePath
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 129
Description
## Problem / current limitation
`RewriteTablePathUtil` / `RewriteTablePathSparkAction.rewriteLocationPrefix(sourcePrefix, targetPrefix)` assume a table's metadata files and data files all live under one common source root that maps 1:1 onto one target root.
In practice this doesn't always hold: a table's metadata can live under a different physical root than its data (different buckets/storage accounts, a `write.metadata.path`/`write.data.path` override that diverges from `location`, or a cross-cloud/cross-DC migration where metadata is staged separately from data). In these cases there is no single `(source, target)` prefix pair that correctly rewrites every path referenced by table metadata (manifest lists, manifests, data files, delete files, statistics files, `write.metadata.path`/`write.data.path` properties, etc).
## Use case
At Datadog we run a cross-datacenter Iceberg table sync job that copies a table's on-disk state (metadata + data) between two datacenters/clouds via `RewriteTablePath`, producing rewritten metadata pointing at the destination. Some of our tables have metadata staged under one root/bucket while the data files live under a separate root/bucket. The existing single-prefix API can't express this without either forcing everything under one prefix (not always possible) or maintaining a private fork of these classes.
## Proposed feature
Generalize the prefix resolution used by `RewriteTablePathUtil`/`RewriteTablePathSparkAction` to accept:
- an optional metadata-specific `(metadataSourcePrefix, metadataTargetPrefix)` pair
- an optional data-specific `(dataSourcePrefix, dataTargetPrefix)` pair
- the existing default `(sourcePrefix, targetPrefix)` pair as a fallback
When rewriting any path, the resolver picks the most specific matching prefix (longest source-prefix match among metadata/data/default), falling back to the default pair — so `rewriteLocationPrefix(source, target)` keeps working unchanged for the existing single-pair case.
We already have a working prototype against Iceberg 1.10.1:
- A `PathPrefixRewrite` resolver with `single(...)` (matches current behavior) and a new `dual(defaultSource, defaultTarget, metaSource, metaTarget, dataSource, dataTarget)` factory, sharing longest-prefix-match resolution across `isKnownPath`, `relativize`, `rewritePath`, `matchedPair`.
- `RewriteTablePathSparkAction` extended with a `rewritePrefixes(PathPrefixRewrite)` action method alongside the existing `rewriteLocationPrefix(String, String)`, with all internal call sites (manifest list, manifest, position/equality delete rewrite, table metadata/property rewrite) switched to use the resolver.
We'd like to contribute this back as a PR if the community is open to the direction — happy to adjust the API shape (naming, whether this lives in `RewriteTablePathUtil` directly vs. a separate strategy object) based on maintainer feedback.
## Query engine
Spark
Contributor guide
Research direction
Start with RewriteTablePathUtil and RewriteTablePathSparkAction, especially rewriteLocationPrefix and the internal manifest, delete, and table metadata/property rewrite call sites. Review the proposed PathPrefixRewrite prototype and its single/dual and longest-prefix resolution behavior. Done means metadata and data paths can use separate optional prefix pairs while the existing single-pair API continues to work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100