[Bug](iceberg) UPDATE/DELETE on ADLS (abfss) tables commits position-delete files with Doris internal s3:// path, permanently breaking reads
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
> **Note on authorship:** this report was researched and written by **Claude Code** (Anthropic's CLI
> coding agent), driven by and reviewed by the account owner. The source analysis, the live
> reproduction, and the cleanup were all executed as described below against a real Doris 4.1.3-rc02
> cluster on a disposable table. Paths, account names and hostnames in the repro have been
> genericised; everything else is verbatim.
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
Searched for `convertToS3Style`, `Failed to create LocationPath`, `delete_pos`, `LocationPath`, and
`iceberg delete file abfss s3` — no existing report.
### Version
- Reproduced on **`doris-4.1.3-rc02-7126cf65d96`** (3 FEs, `SHOW FRONTENDS`).
- Also present on **`master` @ `b58b2c53ff56354c692c2dc7634d724d8780838f`** by code inspection (see *Anything Else?* — the relevant BE writer is byte-identical and the FE lost the field on the delete lane).
- Catalog: Iceberg REST (Lakekeeper 0.13.1), storage **ADLS Gen2 (`abfss://`)**, Iceberg format-version 2, `write.delete.mode=merge-on-read`.
### What's Wrong?
On an Iceberg catalog backed by **ADLS Gen2**, a SQL `UPDATE` or `DELETE` commits a snapshot whose
**position-delete file is recorded in the manifest with Doris's internal `s3://`-normalized path**
instead of the real `abfss://` URI. Every subsequent read of the table then fails permanently:
```
ERROR 1105 (HY000): errCode = 2, detailMessage =
Failed to create LocationPath for location: s3:////data/delete_pos__125613.zstd.parquet
```
`INSERT` on the same table is unaffected — its data files are written with correct `abfss://` paths.
**The same `UPDATE` statement writes a correct `abfss://` data file and a broken `s3://` delete
file**, which isolates the defect to the position-delete lane.
The table is not recoverable through SQL — the bad path is durable Iceberg metadata. It has to be
rolled back to the previous snapshot out-of-band (we use pyiceberg
`manage_snapshots().rollback_to_snapshot(...)`, followed by `REFRESH TABLE`).
**Root cause.** `abfss://@.dfs.core.windows.net/` is deliberately normalized
to `s3:///` for the BE (the account is re-attached BE-side from `azure.account_name`).
That internal form is supposed to stay internal: the FE sends the BE **both** the normalized path
(`output_path`) and the original (`original_output_path`), and the writer is expected to report the
*original* back for the manifest. The **data** writer does this. The **position-delete** writer does
not — it reports its normalized `output_path`, and that string is written straight into the manifest.
On the read side, `LocationPath.findStorageProperties()` then sees scheme `s3` on an Azure-only
catalog. `AzureProperties.getStorageName()` is `"AZURE"`, and the legacy compatibility fallback only
matches a storage config literally named `"s3"` — so nothing matches and construction throws.
Note this is *not* a catalog misconfiguration and *not* the REST catalog's doing: the catalog returns
correct fully-qualified `abfss://` locations (verified on the wire), and the table has no
`write.data.path` / `write.object-storage.path` / `write.folder-storage.path` set. The `s3://` string
does not exist anywhere until Doris synthesises it.
### What You Expected?
`UPDATE` / `DELETE` on an `abfss://`-backed Iceberg table should record position-delete files in the
manifest using the same original `abfss://` URI scheme that data files already use, so the table
remains readable after the commit.
Failing that, Doris should reject the DML rather than commit a snapshot that permanently breaks reads.
### How to Reproduce?
Iceberg REST catalog on ADLS Gen2. Catalog roughly:
```sql
CREATE CATALOG `iceberg_adls` PROPERTIES (
"type" = "iceberg",
"iceberg.catalog.type" = "rest",
"uri" = "http://:8181/catalog",
"warehouse" = "",
"fs.azure.support" = "true",
"azure.account_name" = "",
"azure.account_key" = ""
);
```
Then, on a throwaway table:
```sql
-- 1. create
CREATE TABLE iceberg_adls..zz_probe (
id INT NOT NULL,
val STRING NULL
) ENGINE = iceberg
PROPERTIES (
'write-format' = 'parquet',
'compression-codec' = 'zstd',
'format-version' = '2',
'write.delete.mode' = 'merge-on-read'
);
-- 2. insert
INSERT INTO iceberg_adls..zz_probe VALUES (1,'a'),(2,'b'),(3,'c');
-- 3. reads fine, and every file is abfss://
SELECT * FROM iceberg_adls..zz_probe ORDER BY id;
SELECT content, file_path FROM iceberg_adls..zz_probe$files;
-- content | file_path
-- 0 | abfss://@.dfs.core.windows.net//data/....zstd.parquet
-- 4. one row-level update
UPDATE iceberg_adls..zz_probe SET val='UPDATED' WHERE id = 2;
-- 5. the delete file is s3://, the data files (including the one this UPDATE just wrote) are abfss://
SELECT content, file_path FROM iceberg_adls..zz_probe$files;
```
Observed at step 5:
| content | file_path |
|---|---|
| 0 (data) | `abfss://@.dfs.core.windows.net//data/….zstd.parquet` |
| 0 (data — **written by the UPDATE**) | `abfss://@.dfs.core.windows.net//data/….zstd.parquet` |
| 1 (position delete) | **`s3:////data/delete_pos__125613.zstd.parquet`** |
```sql
-- 6. the table is now permanently unreadable
SELECT * FROM iceberg_adls..zz_probe ORDER BY id;
-- ERROR 1105 (HY000): errCode = 2, detailMessage =
-- Failed to create LocationPath for location: s3:////data/delete_pos__125613.zstd.parquet
```
`EXPLAIN` still succeeds at step 6 — planning does not resolve the delete file, so only execution fails.
We ran this twice on two independent fresh tables; identical result both times. Both were dropped afterwards.
Cluster: 3 FE / 3 BE, `doris-4.1.3-rc02-7126cf65d96`, Iceberg REST catalog (Lakekeeper 0.13.1) over ADLS Gen2 with shared-key auth.
### Anything Else?
**Where it goes wrong, with line references.**
*Normalization (by design).* `AzurePropertyUtils.validateAndNormalizeUri()` — javadoc literally reads
*"@return a normalized `s3://`-style URI"* — calls `convertToS3Style()`, which maps
`abfss://@.dfs.core.windows.net/` → `s3:///`, dropping the
account. Unconditional; the only bypass is `isOneLakeLocation()` (MS Fabric
`*.dfs.fabric.microsoft.com`, returned unchanged). There is no user-facing switch to disable it.
(`fe/fe-core/.../datasource/property/storage/AzurePropertyUtils.java` on `branch-4.1`; FE storage
properties were refactored into `fe/fe-filesystem/fe-filesystem-azure/` on master.)
*The FE hands the BE both forms.* On `branch-4.1`,
`fe/fe-core/src/main/java/org/apache/doris/planner/IcebergTableSink.java:241-243`:
```java
String originalLocation = writeSchemaContext
.map(IcebergWriteSchemaContext::getDataLocation)
.orElseGet(() -> IcebergUtils.dataLocation(icebergTable));
LocationPath locationPath = LocationPath.of(originalLocation, storagePropertiesMap);
tSink.setOutputPath(locationPath.toStorageLocation().toString()); // s3://...
tSink.setOriginalOutputPath(originalLocation); // abfss://...
```
*Data writer uses the original — correct.*
`be/src/exec/sink/writer/iceberg/viceberg_table_writer.cpp:618-646` builds `original_write_path` from
`iceberg_table_sink.original_output_path` and passes it in `WriteInfo`.
*Position-delete writer does not.*
`be/src/exec/sink/writer/iceberg/viceberg_delete_file_writer.cpp` contains **zero** references to
`original_output_path` / `original_write_path` on both `branch-4.1` and `master`:
```cpp
// line 50
_output_path(output_path),
// line 76
io::FileDescription file_description = {.path = _output_path, .fs_name {}};
// line 150 <-- the normalized s3:// path is what gets reported for the manifest
commit_data.__set_file_path(_output_path);
```
*And the FE writes that string verbatim.*
`IcebergWriterHelper.convertToDeleteFiles()` — `String deleteFilePath = commitData.getFilePath(); …
.withPath(deleteFilePath)` (line ~328 on `branch-4.1`, ~326 on master). By contrast the data path at
line ~95/133 receives an already-corrected value from the BE.
**On master the delete lane is missing the plumbing at the FE too.** In
`fe/fe-connector/fe-connector-iceberg/.../IcebergWritePlanProvider.java` (@ `b58b2c53`):
| method | line | sets `originalOutputPath`? |
|---|---|---|
| `buildRewriteSink` | 747-752 | ✅ yes |
| **`buildDeleteSink`** | **816-817** | ❌ **no** — sets `outputPath` + `tableLocation` only |
| `buildMergeSink` | 889-891 | ✅ yes |
**Suggested fix.** Thread `original_output_path` through the delete lane exactly as the data lane
already does — set it in `buildDeleteSink` on the FE, and have `VIcebergDeleteFileWriter` keep the
original alongside `_output_path` and report *that* in `commit_data.__set_file_path(...)`, while still
writing the file through the normalized path. Any storage whose URI is rewritten on the way to the BE
is affected by the same asymmetry, so this is not Azure-specific in principle — Azure is just where
the rewrite is unconditional.
**Not a workaround:** `ALTER TABLE … SET ('write.data.path'='abfss://…')` is rejected
(`Unknown table property: [write.data.path]`), and setting it out-of-band would not help — it only
changes which directory is fed into the normalizer, and the delete writer still reports the
post-normalization value. Adding S3 credentials to the catalog only makes `s3:///…`
resolve to a nonexistent S3 bucket.
Related but **not** a fix for this: #64028 / #64042 (`IcebergUtils.dataLocation` priority chain) —
that only selects *which* directory; all branches still pass through `convertToS3Style()`. We are on a
build that contains it and still reproduce.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
Happy to test a patch against the reproduction above and report back.
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Start with IcebergWritePlanProvider.java, IcebergTableSink.java, viceberg_table_writer.cpp, and viceberg_delete_file_writer.cpp, then compare the data and position-delete paths. Verify the reproduction on an ADLS Gen2 Iceberg table and confirm that UPDATE/DELETE commits retain the original abfss:// location while reads remain successful.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, cpp, java
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100