AnswerDotAI / AnswerDotAI/RAGatouille
Trainer stuck
- Dominant language
- Python
- Stars
- 4k
- Forks
- 276
- PR merge metrics
- No merged PRs in 30d
Description
Hi !
I tried to execute the basic_training notebook on Google collab.
The `trainer.train()` phase is stuck in something an infinite loop for hours after printing only "#> Starting..."
Here is the code that is displayed when I interrupt the execution :
```
[/usr/local/lib/python3.10/dist-packages/ragatouille/RAGTrainer.py](https://localhost:8080/#) in train(self, batch_size, nbits, maxsteps, use_ib_negatives, learning_rate, dim, doc_maxlen, use_relu, warmup_steps, accumsteps)
236 )
237
--> 238 return self.model.train(data_dir=self.data_dir, training_config=training_config)
[/usr/local/lib/python3.10/dist-packages/ragatouille/models/colbert.py](https://localhost:8080/#) in train(self, data_dir, training_config)
450 )
451
--> 452 trainer.train(checkpoint=self.checkpoint)
453
454 def _colbert_score(self, Q, D_padded, D_mask):
[/usr/local/lib/python3.10/dist-packages/colbert/trainer.py](https://localhost:8080/#) in train(self, checkpoint)
29 launcher = Launcher(train)
30
---> 31 self._best_checkpoint_path = launcher.launch(self.config, self.triples, self.queries, self.collection)
32
33
[/usr/local/lib/python3.10/dist-packages/colbert/infra/launcher.py](https://localhost:8080/#) in launch(self, custom_config, *args)
70 # TODO: If the processes crash upon join, raise an exception and don't block on .get() below!
71
---> 72 return_values = sorted([return_value_queue.get() for _ in all_procs])
73 return_values = [val for rank, val in return_values]
74
[/usr/local/lib/python3.10/dist-packages/colbert/infra/launcher.py](https://localhost:8080/#) in (.0)
70 # TODO: If the processes crash upon join, raise an exception and don't block on .get() below!
71
---> 72 return_values = sorted([return_value_queue.get() for _ in all_procs])
73 return_values = [val for rank, val in return_values]
74
[/usr/lib/python3.10/multiprocessing/queues.py](https://localhost:8080/#) in get(self, block, timeout)
101 if block and timeout is None:
102 with self._rlock:
--> 103 res = self._recv_bytes()
104 self._sem.release()
105 else:
[/usr/lib/python3.10/multiprocessing/connection.py](https://localhost:8080/#) in recv_bytes(self, maxlength)
214 if maxlength is not None and maxlength < 0:
215 raise ValueError("negative maxlength")
--> 216 buf = self._recv_bytes(maxlength)
217 if buf is None:
218 self._bad_message_length()
[/usr/lib/python3.10/multiprocessing/connection.py](https://localhost:8080/#) in _recv_bytes(self, maxsize)
412
413 def _recv_bytes(self, maxsize=None):
--> 414 buf = self._recv(4)
415 size, = struct.unpack("!i", buf.getvalue())
416 if size == -1:
[/usr/lib/python3.10/multiprocessing/connection.py](https://localhost:8080/#) in _recv(self, size, read)
377 remaining = size
378 while remaining > 0:
--> 379 chunk = read(handle, remaining)
380 n = len(chunk)
381 if n == 0:
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by running the basic_training notebook and inspect RAGTrainer.py at train(), then follow the call into colbert/trainer.py and colbert/infra/launcher.py. Determine why launcher.launch waits on return_value_queue.get() after the training processes start; done means the notebook completes training or reports a clear failure instead of remaining blocked.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100