Error while passing Multiple Datasets during Training/Fine-Tuning
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13.7k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
Hi there. I was able to generate Dataset (or you can say, run Dataset Preparation steps) however, I am receiving the following error, when I'm trying to pass multiple datasets in the training command, as suggested by you, i.e.
python train.py \
--config configs/generation/slat_flow_img_dit_L_64l8p2_fp16.json \
--output_dir outputs/slat_flow_img_dit_L_64l8p2_fp16_1node \
--data_dir datasets/ABO, datasets/HSSD \
--profile
It is giving me the following error:
Error: BasicTrainer.run_step() missing 1 required positional argument: 'data_list'
Please confirm if it supports multiple dataset or not, because otherwise it's useless to keep separate datasets and one has to combine everything during data preparation. I tried some changes, but the issue may require deep digging and changes in the whole codebase (responsible for data loading and training).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with train.py and BasicTrainer.run_step, then trace how the --data_dir value is parsed and passed as data_list. Reproduce the command using the listed config and dataset directories; done means multiple datasets are accepted without the missing data_list error and training starts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100