ChenRocks / ChenRocks/UNITER

Pretraining from scratch on small data

Open
#74 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
799
Forks
111
PR merge metrics
No merged PRs in 30d

Description

Hello.
I want to pre-train myself.

I try running the CMD here:
https://github.com/ChenRocks/UNITER#pre-tranining

```
horovodrun -np 8 python pretrain.py --config config/pretrain-indomain-base-8gpu.json \
--output_dir $PRETRAIN_EXP
```

(after launching the docker)

But I receieve:
`[1,0]:RuntimeError: cuda runtime error (35) : CUDA driver version is insufficient for CUDA runtime version at ../torch/csrc/cuda/Module.cpp:33`

I use Cuda 10.2.

Do you know how can I solve it?

I addition, I would like to ask how long should the pretraining take?

Thank you

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the README's pre-training instructions and the `pretrain.py` command using `config/pretrain-indomain-base-8gpu.json`; inspect how the Docker environment selects its CUDA runtime and compare that with the reported CUDA 10.2 setup. The issue is resolved when the command runs without the CUDA driver/runtime error and the expected pretraining duration is documented or explained.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python, pytorch
Domain
devops, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.