apache / apache/iceberg

Azure: ADLSFileIO.listPrefix returns relative paths instead of fully-qualified URIs

Open Beginner friendly
#17,247 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 16h
Merged PRs (30d)
129

Description

**Apache Iceberg version**

main @ 5f3d3c560

**Query engine**

Spark / Flink (only on the opt-in prefix-listing path; engine-agnostic otherwise)

**Please describe the bug**

`ADLSFileIO.listPrefix()` returns each file's location as a path relative to the filesystem
container root (e.g. `dir/file`) instead of a fully-qualified URI — `new FileInfo(pathItem.getName(), ...)`
at ADLSFileIO.java:230. The siblings return full URIs: `S3FileIO` builds `s3://bucket/key`
(S3FileIO.java:348) and `GCSFileIO` builds `gs://bucket/name` (GCSFileIO.java:302).

**Steps to reproduce**

`FileSystemWalker.listDirRecursivelyWithFileIO()` (core) assumes each `FileInfo.location()` is a
full URI and feeds it back through `ADLSFileIO`. A relative path fails `ADLSLocation`'s
`URI_PATTERN` (`^(abfss?|wasbs?)://...`), throwing `ValidationException: Invalid ADLS URI: dir/file`.
This is reachable from Spark `DeleteOrphanFilesSparkAction` with `usePrefixListing(true)` (opt-in,
default false) and from Flink maintenance orphan-file detection: the listed path does not match the
metadata URI, so the action fails with a `BulkDeletionFailureException`. No live files are deleted,
since the same validation blocks the delete.

Expected: `listPrefix` returns `abfss://container@account.dfs.core.windows.net/dir/file`, matching
S3 and GCS. The gap dates back to the initial ADLS FileIO (#8303, 2023).

**Additional context**

N/A

Contributor guide

Open the contributing guide

Research direction

Start at ADLSFileIO.java:230 and compare its listPrefix location handling with S3FileIO.java:348 and GCSFileIO.java:302. Trace FileSystemWalker.listDirRecursivelyWithFileIO() and the Spark or Flink orphan-file paths, then run the relevant existing tests. Done means listed locations are fully-qualified ADLS URIs and prefix-based orphan-file processing no longer rejects them.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, java
Domain
cloud
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.