[Bug] Incorrect local file path in HfUploader
Open
Nobody has claimed this yet.
bug
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 206
- PR merge metrics
- No merged PRs in 30d
Description
To reproduce
Steps to reproduce the behavior:
- Setup mosaicml-streaming from the main branch
pip install git+https://github.com/mosaicml/streaming.git - Run sample code:
from streaming import MDSWriter
#replace with the path to the hf datasets repo that you want to upload to.
out = "hf://datasets/NagaSaiAbhinay/MDSTest/"
columns = {
"id": "int",
"latents": "ndarray"
}
## HF_TOKEN is set in the env variables.
import torch
with MDSWriter(out=out, columns=columns, exist_ok=True, progress_bar=True) as writer:
for i in range(100):
sample = {
"id": i,
"latents": torch.randn((100,100), dtype=torch.float16).numpy()
}
writer.write(sample)
Expected behavior
Successful upload of local temp shards to huggingface datasets repo.
Observed behavior
Failure with traceback
No files have been modified since last commit. Skipping to prevent empty commit.
Exception in writer thread: [Errno 2] No such file or directory: 'index.json'
No files have been modified since last commit. Skipping to prevent empty commit.
Exception in writer thread: [Errno 2] No such file or directory: 'shard.00000.mds'
---------------------------------------------------------------------------
Exception Traceback (most recent call last)
Cell In[7], line 3
1 get_ipython().system('rm -rf ./test')
2 import torch
----> 3 with MDSWriter(out=out, columns=columns, exist_ok=True, progress_bar=True) as writer:
4 for i in range(100):
5 sample = {
6 "id": i,
7 "latents": torch.randn((100,100), dtype=torch.float16).numpy()
8 }
File [/usr/local/lib/python3.11/dist-packages/streaming/base/format/base/writer.py:358](https://os0uhq407m4ki7-8888.proxy.runpod.net/lab/tree/workspace/usr/local/lib/python3.11/dist-packages/streaming/base/format/base/writer.py#line=357), in Writer.__exit__(self, exc_type, exc, traceback)
349 def __exit__(self, exc_type: Optional[type[BaseException]], exc: Optional[BaseException],
350 traceback: Optional[TracebackType]) -> None:
351 """Exit context manager.
352
353 Args:
(...)
356 traceback (TracebackType, optional): Traceback.
357 """
--> 358 self.finish()
File [/usr/local/lib/python3.11/dist-packages/streaming/base/format/base/writer.py:313](https://os0uhq407m4ki7-8888.proxy.runpod.net/lab/tree/workspace/usr/local/lib/python3.11/dist-packages/streaming/base/format/base/writer.py#line=312), in Writer.finish(self)
311 # Final check, in case error occurred in an upload during shutdown
312 if self.event.is_set():
--> 313 raise Exception('One of the threads failed. Check other traceback for more ' +
314 'details.')
Exception: One of the threads failed. Check other traceback for more details.
Additional context
The issue is that the HfUploader.upload_file is not appending the temp directory path where shards are stored to the filenames.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Locate HfUploader.upload_file and trace how the temporary shard directory is passed to uploaded filenames. Reproduce the issue with the MDSWriter example and inspect the writer-thread traceback. Done means the local index and shard files upload successfully to the Hugging Face dataset without missing-file errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python, pytorch
- Domain
- cloud, data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100