apache / apache/iceberg

Support separate metadata/data location prefixes in RewriteTablePath

Open
#17,794 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 16h
Merged PRs (30d)
129

Description

## Problem / current limitation
`RewriteTablePathUtil` / `RewriteTablePathSparkAction.rewriteLocationPrefix(sourcePrefix, targetPrefix)` assume a table's metadata files and data files all live under one common source root that maps 1:1 onto one target root.

In practice this doesn't always hold: a table's metadata can live under a different physical root than its data (different buckets/storage accounts, a `write.metadata.path`/`write.data.path` override that diverges from `location`, or a cross-cloud/cross-DC migration where metadata is staged separately from data). In these cases there is no single `(source, target)` prefix pair that correctly rewrites every path referenced by table metadata (manifest lists, manifests, data files, delete files, statistics files, `write.metadata.path`/`write.data.path` properties, etc).

## Use case
At Datadog we run a cross-datacenter Iceberg table sync job that copies a table's on-disk state (metadata + data) between two datacenters/clouds via `RewriteTablePath`, producing rewritten metadata pointing at the destination. Some of our tables have metadata staged under one root/bucket while the data files live under a separate root/bucket. The existing single-prefix API can't express this without either forcing everything under one prefix (not always possible) or maintaining a private fork of these classes.

## Proposed feature
Generalize the prefix resolution used by `RewriteTablePathUtil`/`RewriteTablePathSparkAction` to accept:
- an optional metadata-specific `(metadataSourcePrefix, metadataTargetPrefix)` pair
- an optional data-specific `(dataSourcePrefix, dataTargetPrefix)` pair
- the existing default `(sourcePrefix, targetPrefix)` pair as a fallback

When rewriting any path, the resolver picks the most specific matching prefix (longest source-prefix match among metadata/data/default), falling back to the default pair — so `rewriteLocationPrefix(source, target)` keeps working unchanged for the existing single-pair case.

We already have a working prototype against Iceberg 1.10.1:
- A `PathPrefixRewrite` resolver with `single(...)` (matches current behavior) and a new `dual(defaultSource, defaultTarget, metaSource, metaTarget, dataSource, dataTarget)` factory, sharing longest-prefix-match resolution across `isKnownPath`, `relativize`, `rewritePath`, `matchedPair`.
- `RewriteTablePathSparkAction` extended with a `rewritePrefixes(PathPrefixRewrite)` action method alongside the existing `rewriteLocationPrefix(String, String)`, with all internal call sites (manifest list, manifest, position/equality delete rewrite, table metadata/property rewrite) switched to use the resolver.

We'd like to contribute this back as a PR if the community is open to the direction — happy to adjust the API shape (naming, whether this lives in `RewriteTablePathUtil` directly vs. a separate strategy object) based on maintainer feedback.

## Query engine
Spark

Contributor guide

Open the contributing guide

Research direction

Start with RewriteTablePathUtil and RewriteTablePathSparkAction, especially rewriteLocationPrefix and the internal manifest, delete, and table metadata/property rewrite call sites. Review the proposed PathPrefixRewrite prototype and its single/dual and longest-prefix resolution behavior. Done means metadata and data paths can use separate optional prefix pairs while the existing single-pair API continues to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.