apache / apache/incubator-xtable

Absolute paths in the Delta log are treated as relative, silently dropping rows from the converted table

Open Beginner friendly
#907 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
1.2k
Forks
212
Avg merge
4d 9h
Merged PRs (30d)
16

Description

### Describe the bug

When a Delta commit references a data file by absolute path, XTable concatenates that path onto
the table base instead of using it. The resulting Iceberg manifest entry points at a location
that does not exist. The entry carries `record_count=0` and is pruned during scan planning, so
no reader ever tries to open it — the sync reports success and the converted table silently
returns fewer rows than the source, with no error anywhere.

The Delta protocol permits absolute paths. From `PROTOCOL.md`, Add File action:

> A relative path to a data file from the root of the table **or an absolute path to a file**
> that should be added to the table.

The same wording appears for the Remove File action.

### Root cause

`DeltaActionsConverter.getFullPathToFile` is the whole of the resolution logic:

```java
static String getFullPathToFile(String tableBasePath, String dataFilePath) {
if (dataFilePath.startsWith(tableBasePath)) {
return dataFilePath;
}
return tableBasePath + Path.SEPARATOR + dataFilePath;
}
```

`startsWith` is doing prefix matching where path containment is meant, so any absolute path not
lexically under the table base is treated as relative. Calling the method directly:

| tableBasePath | dataFilePath | result |
|---|---|---|
| `s3a://bucket/tab` | `part-0.parquet` | ok |
| `s3a://bucket/tab` | `s3a://bucket/tab/part-0.parquet` | ok |
| `s3a://bucket/tab` | `s3://bucket/tab/part-0.parquet` | `s3a://bucket/tab/s3://bucket/tab/part-0.parquet` |
| `s3a://bucket/tab` | `s3a://other-bucket/x/part-0.parquet` | `s3a://bucket/tab/s3a://other-bucket/x/part-0.parquet` |
| `abfss://c@a.dfs.core.windows.net/tab` | `gs://bucket/t/part-0.parquet` | concatenated |
| `s3a://bucket/tab` | `file:///local/t/part-0.parquet` | concatenated |

Note the third and fourth rows are the same defect, not two: a scheme alias (`s3://` against an
`s3a://` base) and a same-scheme different-location path both fail the prefix test. The second
shape is what a shallow clone produces.

### To Reproduce

1. Write a Delta table. Write a parquet file somewhere outside it.
2. Add that file to the table by absolute URI. With `deltalake` 1.6.3:

```python
from deltalake import DeltaTable
from deltalake.transaction import AddAction

dt = DeltaTable(table_path)
dt.create_write_transaction(
[AddAction(path="file:///abs/path/outside.parquet", size=..., partition_values={},
modification_time=..., data_change=True, stats=...)],
"append", schema, partition_by=[])
```

The absolute URI is persisted verbatim — no validation, no rewriting.

3. Run `RunSync` DELTA → ICEBERG against the table.

Observed, with a 2-file / 5-row source:

```
Sync is successful for the following formats ICEBERG exit 0

manifest entries:
status=ADDED records=0
file:/.../table/file:///.../elsewhere/part-00000-....parquet <- base + absolute URI
status=ADDED records=3
file:/.../table/part-00000-....parquet

snapshot summary: total-data-files=2 total-records=3
```

PyIceberg's `plan_files` returns only the second entry — the mangled one is pruned before any
I/O is attempted — and the scan returns **3 rows**. The two rows behind the absolute path are
absent, with no error at any point.

### Expected behavior

An absolute path in the Delta log should be used as-is. If XTable cannot support a data file
outside the table root, it should fail rather than emit a manifest entry pointing at a
concatenated path.

### On how reachable this is

Being straight about the evidence, since it affects how much this matters:

- The protocol permits absolute paths, quoted above.
- delta-spark passes such an entry through to XTable — that is how the conversion sees it.
- delta-rs persists one through its public API without complaint, shown above. Its own writers
emit relative paths, so it is not a spontaneous producer, but it is a seam.

What I cannot show is a product that emits absolute paths by default. Databricks shallow clone
is the commonly cited producer and I have no way to test it. So this may be a latent defect
rather than one users are hitting today — but the input is legal, the failure is silent, and the
fix is small.

### Related, but not the same

#323 covers the reverse direction — the Delta *target* converting absolute paths to relative in
`DeltaDataFileUpdatesExtractor`, and asks for that conversion to be customisable. This report is
about the Delta *source* resolving a logged path in `DeltaActionsConverter`. Same subject area,
different code path, and neither fix would address the other. #171 and #230 are also path-related
but concern the table base path and the Delta target respectively.

### Environment

- XTable `main`, version 0.5.0-SNAPSHOT, bundled jar, Corretto 11
- `deltalake` 1.6.3 / PyArrow 25.0.1 wrote the fixture; PyIceberg read the result

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

*This issue was created with AI assistance.*

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at DeltaActionsConverter.getFullPathToFile, the resolution entry point identified in the issue, and compare its behavior with the listed relative and absolute URI cases. Run the DELTA → ICEBERG sync reproduction and verify that absolute paths are preserved and the converted table retains all source rows without a mangled manifest location.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.