apache / apache/hudi-rs

Windows path separator bug: library uses backslash instead of forward slash, causing issues for cloud and POSIX APIs

Open
#445 1 comment 0 reactions 1 assignee Claimed by @yunchipang View on GitHub
bug
Dominant language
Rust
Stars
279
Forks
67
Avg merge
3d 1h
Merged PRs (30d)
40

Description

### Is there an existing issue for this?

- [x] I have searched the existing issues

### Description of the bug

On Windows, the library uses `PathBuf` for joining paths, which results in paths with backslashes (`\`). This causes problems when interacting with cloud storage APIs or POSIX-style APIs that expect forward slashes (`/`) as the path separator. As a result, file operations may fail, or the generated paths may not be accepted by external systems. There is no documented workaround or environment variable to force forward slash normalization for cross-platform compatibility.

### Steps To Reproduce

1. Use python hudi to access some hudi table in S3 on Windows.

2. Run example code:
```python
from hudi import HudiTableBuilder
import pyarrow as pa

hudi_table = HudiTableBuilder.from_base_uri("s3://path/to_your_hudi/").build()
batches = hudi_table.read_snapshot(filters=[("city", "=", "san_francisco")])

# convert to PyArrow table
arrow_table = pa.Table.from_batches(batches)
result = arrow_table.select(["rider", "city", "ts", "fare"])
print(result)
```
3. Observe s3 error: path to .hoodie snapshot will be concatenated with backslash

### Expected behavior

Paths should always use forward slashes (`/`) when interacting with external APIs, regardless of the underlying OS. There should be a documented way (e.g., environment variable or API option) to normalize paths for cross-platform compatibility.

### Screenshots / Logs

Storage error: Object at location
my_table/.hoodie\20250909105111479.deltacommit not found: Client error with status 404 Not Found:
NoSuchKeyThe specified key does not
exist.my_table/.hoodie\20250909105111479.deltacommitXXXXYYYY

### Software information

- Operating system: Windows
- Project version: 0.4.0

### Additional context

This bug can cause major interoperability problems for users working on Windows, especially with cloud storage (S3, GCS, etc.). Please consider adding a normalization step or a config option/environment variable to always use POSIX-style paths when interacting with external systems.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.