microsoft / microsoft/XPretrain
Error in finetuning
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 511
- Forks
- 35
- PR merge metrics
- No merged PRs in 30d
Description
When running the command inside the docker image for finetuning LF-VILA, following error is created,
root@8dccc81930c3:/LF-VILA# deepspeed src/tasks/run_video_classification.py --distributed --blob_mount_dir /blob_mount --config $CONFIG_PATH --deepspeed
[2023-10-17 11:11:02,765] [WARNING] [runner.py:132:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
Traceback (most recent call last):
File "/usr/local/bin/deepspeed", line 6, in
main()
File "/usr/local/lib/python3.8/dist-packages/deepspeed/launcher/runner.py", line 308, in main
raise RuntimeError("Unable to proceed, no GPU resources available")
RuntimeError: Unable to proceed, no GPU resources available
Please note that my device has GPUs available and cuda and torch are correctly installed.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the documented deepspeed command for src/tasks/run_video_classification.py inside the Docker image and reproduce the reported error. Check why DeepSpeed cannot see the available GPU despite the reported CUDA and PyTorch installations. Done means the command detects local GPU resources and proceeds past launcher initialization.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100