mosaicml / mosaicml/streaming

[Bug] Incorrect local file path in HfUploader

Open
#875 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
1.6k
Forks
206
PR merge metrics
No merged PRs in 30d

Description

To reproduce

Steps to reproduce the behavior:

  1. Setup mosaicml-streaming from the main branch
    pip install git+https://github.com/mosaicml/streaming.git
  2. Run sample code:
from streaming import MDSWriter
#replace with the path to the hf datasets repo that you want to upload to.
out = "hf://datasets/NagaSaiAbhinay/MDSTest/"  
columns = {
    "id": "int",
    "latents": "ndarray"
}
## HF_TOKEN is set in the env variables.
import torch
with MDSWriter(out=out, columns=columns, exist_ok=True, progress_bar=True) as writer:
    for i in range(100):
        sample = {
            "id": i,
            "latents": torch.randn((100,100), dtype=torch.float16).numpy()
        }
        writer.write(sample)

Expected behavior

Successful upload of local temp shards to huggingface datasets repo.

Observed behavior

Failure with traceback

No files have been modified since last commit. Skipping to prevent empty commit.
Exception in writer thread: [Errno 2] No such file or directory: 'index.json'
No files have been modified since last commit. Skipping to prevent empty commit.
Exception in writer thread: [Errno 2] No such file or directory: 'shard.00000.mds'
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
Cell In[7], line 3
      1 get_ipython().system('rm -rf ./test')
      2 import torch
----> 3 with MDSWriter(out=out, columns=columns, exist_ok=True, progress_bar=True) as writer:
      4     for i in range(100):
      5         sample = {
      6             "id": i,
      7             "latents": torch.randn((100,100), dtype=torch.float16).numpy()
      8         }

File [/usr/local/lib/python3.11/dist-packages/streaming/base/format/base/writer.py:358](https://os0uhq407m4ki7-8888.proxy.runpod.net/lab/tree/workspace/usr/local/lib/python3.11/dist-packages/streaming/base/format/base/writer.py#line=357), in Writer.__exit__(self, exc_type, exc, traceback)
    349 def __exit__(self, exc_type: Optional[type[BaseException]], exc: Optional[BaseException],
    350              traceback: Optional[TracebackType]) -> None:
    351     """Exit context manager.
    352 
    353     Args:
   (...)
    356         traceback (TracebackType, optional): Traceback.
    357     """
--> 358     self.finish()

File [/usr/local/lib/python3.11/dist-packages/streaming/base/format/base/writer.py:313](https://os0uhq407m4ki7-8888.proxy.runpod.net/lab/tree/workspace/usr/local/lib/python3.11/dist-packages/streaming/base/format/base/writer.py#line=312), in Writer.finish(self)
    311 # Final check, in case error occurred in an upload during shutdown
    312 if self.event.is_set():
--> 313     raise Exception('One of the threads failed. Check other traceback for more ' +
    314                     'details.')

Exception: One of the threads failed. Check other traceback for more details.

Additional context

The issue is that the HfUploader.upload_file is not appending the temp directory path where shards are stored to the filenames.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Locate HfUploader.upload_file and trace how the temporary shard directory is passed to uploaded filenames. Reproduce the issue with the MDSWriter example and inspect the writer-thread traceback. Done means the local index and shard files upload successfully to the Hugging Face dataset without missing-file errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python, pytorch
Domain
cloud, data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.