delta-io / delta-io/delta-kernel-rs
`https://` URLs are broken on Azure
- Dominant language
- Rust
- Stars
- 361
- Forks
- 214
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 87
Description
> [!IMPORTANT]
> If you are running across this issue since you're attempting to read from `https://` URL in azure, workaround is simple: just translate to `abfss://` URL. See [microsoft docs] for the translation.
[microsoft docs]: https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction-abfs-uri
while I was implementing #1006 I actually had to do some hacking on our filesystem (LIST) implementation to get things to work when pointing our read-table-multi-threaded example at azure.
at first, i was getting empty listing and 404 for objects. My table path was something like:
```
https://SOMERESOURCEGROUP.blob.core.windows.net/CONTAINER/table_name
```
and i noticed that in our LIST code (default engine `list_from` impl) we would list with:
```
offset: CONTAINER/table_name/_delta_log/00000000000000000000
prefix: CONTAINER/table_name/_delta_log
```
when instead we should just list the part _within_ the container. in order to work around this I had to inject code to drop the first part of the offset/prefix:
```rust
// remove prefix "CONTAINER/"
let prefix = Path::from_iter(prefix.parts().skip(1));
let offset = Path::from_iter(offset.parts().skip(1));
```
so then we would list with
```
offset: table_name/_delta_log/00000000000000000000
prefix: table_name/_delta_log
```
then everything worked.
this begs the question: is listing just broken on Azure right now? are others able to repro?
my repro was just with:
```bash
# inside kernel/src/examples/read-table-multi-threaded
cargo r --release https://SOMERESOURCEGROUP.blob.core.windows.net/CONTAINER/table_name --public
```
Contributor guide
Assessment
This issue has not been assessed yet.