delta-io / delta-io/delta-kernel-rs

`https://` URLs are broken on Azure

Open
#1,007 4 comments 0 reactions 2 assignees Claimed by @rtyler View on GitHub
Dominant language
Rust
Stars
361
Forks
214
Avg merge
2d 17h
Merged PRs (30d)
87

Description

> [!IMPORTANT]
> If you are running across this issue since you're attempting to read from `https://` URL in azure, workaround is simple: just translate to `abfss://` URL. See [microsoft docs] for the translation.

[microsoft docs]: https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction-abfs-uri

while I was implementing #1006 I actually had to do some hacking on our filesystem (LIST) implementation to get things to work when pointing our read-table-multi-threaded example at azure.

at first, i was getting empty listing and 404 for objects. My table path was something like:
```
https://SOMERESOURCEGROUP.blob.core.windows.net/CONTAINER/table_name
```

and i noticed that in our LIST code (default engine `list_from` impl) we would list with:
```
offset: CONTAINER/table_name/_delta_log/00000000000000000000
prefix: CONTAINER/table_name/_delta_log
```
when instead we should just list the part _within_ the container. in order to work around this I had to inject code to drop the first part of the offset/prefix:
```rust
// remove prefix "CONTAINER/"
let prefix = Path::from_iter(prefix.parts().skip(1));
let offset = Path::from_iter(offset.parts().skip(1));
```
so then we would list with
```
offset: table_name/_delta_log/00000000000000000000
prefix: table_name/_delta_log
```
then everything worked.

this begs the question: is listing just broken on Azure right now? are others able to repro?

my repro was just with:
```bash
# inside kernel/src/examples/read-table-multi-threaded
cargo r --release https://SOMERESOURCEGROUP.blob.core.windows.net/CONTAINER/table_name --public
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.