modelscope / modelscope/ms-swift
如何查看查看启训练任务后，训练集预加载打乱切分后的数据？
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
我在训练期间，发现在某几个step的loss出现异常，因此想查看这几个step对应的训练数据是哪几条。但是我目前还没有找到可以查看或者保存每个step对应batch的功能。
我先将训练参数设置为了 --load_from_cache_file false
然后启动了训练任务，但是训练结束后， 我去/root/.cache/modelscope/hub/datasets/路径下查看，发现里面保存了几个以随机码保存的文件，文件格式是.arrow类型。我不知道这些文件和哪个训练任务时对应的、如何正常打开该文件（用pyarrow库打不开）、以及这里面文件中的数据和step的对应关系是什么。
请问可以解答一下么
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names --load_from_cache_file false and the cache directory /root/.cache/modelscope/hub/datasets/, where randomized .arrow files appear. Start by tracing the training entry point that handles dataset loading, shuffling, splitting, and batch creation. Done means providing a reproducible way to identify or save the records for each step and explaining how to open the resulting cache files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100