flyteorg / flyteorg/flyte

# Fix Remote URI Corruption and Leading Slash Truncation in `storage.join()`

Open
#7,999 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
7.5k
Forks
886
Avg merge
1d 12h
Merged PRs (30d)
120

Description

# Fix Remote URI Corruption and Leading Slash Truncation in `storage.join()`

## Background & Problem
In `src/flyte/storage/_storage.py`:
```python
def join(*paths: str) -> str:
"""
Join multiple paths together. This is a wrapper around os.path.join.
# TODO replace with proper join with fsspec root etc

Args:
paths: Paths to be joined.
"""
return str(os.path.join(*paths))
```

1. **Windows Path Corruption for Cloud URIs**: On Windows, `os.path.join("s3://bucket", "prefix", "file.txt")` uses `\` as separator, outputting `"s3://bucket\\prefix\\file.txt"`. Cloud storage services and object stores reject backslashes in keys.
2. **Subpaths with Leading Slash Wipe Out Prefix**: In Python, `os.path.join` treats any argument beginning with `/` (e.g. `storage.join("s3://bucket/dir", "/subfile.txt")`) as an absolute root path and discards all preceding components, returning `"/subfile.txt"`.
3. **Empty arguments**: Calling `storage.join()` with empty args should return `""` cleanly.

## User Review Required
> [!NOTE]
> For remote paths (checked via `is_remote(paths[0])`), components will be joined using forward slashes (`/`), stripping redundant slashes from child components so prefixes are never dropped. For local paths, standard `os.path.join` behavior is preserved.

## Proposed Changes

### Core Library
#### [MODIFY] [src/flyte/storage/_storage.py](file:///c:/Users/Aayush%20Shankar/OneDrive/Desktop/flyte-sdk/flyte-sdk/src/flyte/storage/_storage.py)
- Update `join(*paths: str) -> str` to check if `paths[0]` is a remote URI (using `is_remote(paths[0])`).
- If remote: join using forward slashes `/`, stripping leading and trailing `/` from inner segments, preserving the protocol scheme.
- If local: use `os.path.join(*paths)`.

### Tests
#### [MODIFY] [tests/internal/storage/test_storage.py](file:///c:/Users/Aayush%20Shankar/OneDrive/Desktop/flyte-sdk/flyte-sdk/tests/internal/storage/test_storage.py)
- Add `test_storage_join()` test covering:
- S3 / GCS / ABFS / Flyte remote URIs with multiple path components.
- Remote URIs where subpaths contain leading slashes (`/sub/file.txt`).
- Local relative and absolute paths.
- Empty call `storage.join()`.

## Verification Plan
### Automated Tests
- Run Python verification tests exercising `storage.join` across remote URIs, Windows path styles, and leading-slash subpaths.

Contributor guide

Open the contributing guide

Research direction

Start in src/flyte/storage/_storage.py by reading storage.join() and the existing is_remote() behavior. Add or update coverage in tests/internal/storage/test_storage.py for remote URI components, leading slashes, local paths, and an empty call; the work is done when those cases pass without changing local os.path.join behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.