Azure: ADLSFileIO.listPrefix returns relative paths instead of fully-qualified URIs
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 129
Description
**Apache Iceberg version**
main @ 5f3d3c560
**Query engine**
Spark / Flink (only on the opt-in prefix-listing path; engine-agnostic otherwise)
**Please describe the bug**
`ADLSFileIO.listPrefix()` returns each file's location as a path relative to the filesystem
container root (e.g. `dir/file`) instead of a fully-qualified URI — `new FileInfo(pathItem.getName(), ...)`
at ADLSFileIO.java:230. The siblings return full URIs: `S3FileIO` builds `s3://bucket/key`
(S3FileIO.java:348) and `GCSFileIO` builds `gs://bucket/name` (GCSFileIO.java:302).
**Steps to reproduce**
`FileSystemWalker.listDirRecursivelyWithFileIO()` (core) assumes each `FileInfo.location()` is a
full URI and feeds it back through `ADLSFileIO`. A relative path fails `ADLSLocation`'s
`URI_PATTERN` (`^(abfss?|wasbs?)://...`), throwing `ValidationException: Invalid ADLS URI: dir/file`.
This is reachable from Spark `DeleteOrphanFilesSparkAction` with `usePrefixListing(true)` (opt-in,
default false) and from Flink maintenance orphan-file detection: the listed path does not match the
metadata URI, so the action fails with a `BulkDeletionFailureException`. No live files are deleted,
since the same validation blocks the delete.
Expected: `listPrefix` returns `abfss://container@account.dfs.core.windows.net/dir/file`, matching
S3 and GCS. The gap dates back to the initial ADLS FileIO (#8303, 2023).
**Additional context**
N/A
Contributor guide
Research direction
Start at ADLSFileIO.java:230 and compare its listPrefix location handling with S3FileIO.java:348 and GCSFileIO.java:302. Trace FileSystemWalker.listDirRecursivelyWithFileIO() and the Spark or Flink orphan-file paths, then run the relevant existing tests. Done means listed locations are fully-qualified ADLS URIs and prefix-based orphan-file processing no longer rejects them.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, java
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100