mosaicml / mosaicml/streaming

Windows absolute paths with drive letters incorrectly parsed as cloud provider schemes

Open
#960 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
1.6k
Forks
206
PR merge metrics
No merged PRs in 30d

Description

Environment

  • OS: Windows 10
  • Hardware (GPU, or instance type): Local machine
    streaming library version: 0.13.0

To reproduce

Steps to reproduce the behavior:

  1. On a Windows system, attempt to create an MDSWriter with an absolute Windows path:
from streaming import MDSWriter

columns = {
    'filename': 'str',
    'data': 'ndarray:float32'
}

# This fails on Windows with absolute paths
output_path = r"D:/test"

with MDSWriter(out=output_path, columns=columns) as writer:
    pass
  1. Observe the error:
ValueError: Invalid Cloud provider prefix: d.

Expected behavior

MDSWriter should accept Windows absolute paths (e.g., D:\path\to\dir or D:/path/to/dir) as valid local filesystem paths, similar to how it handles Unix absolute paths (e.g., /path/to/dir).

Root Cause

The issue is in streaming/base/storage/upload.py in the CloudUploader.get() method:

obj = urllib.parse.urlparse(out) if isinstance(out, str) else urllib.parse.urlparse(out[1])
provider_prefix = obj.scheme

When urllib.parse.urlparse() is called on a Windows path like D:/path/to/dir, it incorrectly interprets the drive letter as a URL scheme:

>>> import urllib.parse
>>> urllib.parse.urlparse("D:/test")
ParseResult(scheme='d', netloc='', path='/test', params='', query='', fragment='')

The drive letter D: is parsed as scheme='d', which then fails the validation since 'd' is not in the UPLOADERS dictionary.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in streaming/base/storage/upload.py at CloudUploader.get(), then reproduce the issue with the provided MDSWriter example using a Windows path such as D:/test. Verify that Windows absolute paths are treated as local filesystem paths rather than cloud provider schemes, while the existing provider-prefix handling remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, cloud
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.