AnswerDotAI / AnswerDotAI/RAGatouille

Trainer stuck

Open
#210 8 comments 0 reactions 0 assignees View on GitHub
bug Upstream ColBERT Interaction
Dominant language
Python
Stars
4k
Forks
276
PR merge metrics
No merged PRs in 30d

Description

Hi !
I tried to execute the basic_training notebook on Google collab.
The `trainer.train()` phase is stuck in something an infinite loop for hours after printing only "#> Starting..."
Here is the code that is displayed when I interrupt the execution :
```
[/usr/local/lib/python3.10/dist-packages/ragatouille/RAGTrainer.py](https://localhost:8080/#) in train(self, batch_size, nbits, maxsteps, use_ib_negatives, learning_rate, dim, doc_maxlen, use_relu, warmup_steps, accumsteps)
236 )
237
--> 238 return self.model.train(data_dir=self.data_dir, training_config=training_config)

[/usr/local/lib/python3.10/dist-packages/ragatouille/models/colbert.py](https://localhost:8080/#) in train(self, data_dir, training_config)
450 )
451
--> 452 trainer.train(checkpoint=self.checkpoint)
453
454 def _colbert_score(self, Q, D_padded, D_mask):

[/usr/local/lib/python3.10/dist-packages/colbert/trainer.py](https://localhost:8080/#) in train(self, checkpoint)
29 launcher = Launcher(train)
30
---> 31 self._best_checkpoint_path = launcher.launch(self.config, self.triples, self.queries, self.collection)
32
33

[/usr/local/lib/python3.10/dist-packages/colbert/infra/launcher.py](https://localhost:8080/#) in launch(self, custom_config, *args)
70 # TODO: If the processes crash upon join, raise an exception and don't block on .get() below!
71
---> 72 return_values = sorted([return_value_queue.get() for _ in all_procs])
73 return_values = [val for rank, val in return_values]
74

[/usr/local/lib/python3.10/dist-packages/colbert/infra/launcher.py](https://localhost:8080/#) in (.0)
70 # TODO: If the processes crash upon join, raise an exception and don't block on .get() below!
71
---> 72 return_values = sorted([return_value_queue.get() for _ in all_procs])
73 return_values = [val for rank, val in return_values]
74

[/usr/lib/python3.10/multiprocessing/queues.py](https://localhost:8080/#) in get(self, block, timeout)
101 if block and timeout is None:
102 with self._rlock:
--> 103 res = self._recv_bytes()
104 self._sem.release()
105 else:

[/usr/lib/python3.10/multiprocessing/connection.py](https://localhost:8080/#) in recv_bytes(self, maxlength)
214 if maxlength is not None and maxlength < 0:
215 raise ValueError("negative maxlength")
--> 216 buf = self._recv_bytes(maxlength)
217 if buf is None:
218 self._bad_message_length()

[/usr/lib/python3.10/multiprocessing/connection.py](https://localhost:8080/#) in _recv_bytes(self, maxsize)
412
413 def _recv_bytes(self, maxsize=None):
--> 414 buf = self._recv(4)
415 size, = struct.unpack("!i", buf.getvalue())
416 if size == -1:

[/usr/lib/python3.10/multiprocessing/connection.py](https://localhost:8080/#) in _recv(self, size, read)
377 remaining = size
378 while remaining > 0:
--> 379 chunk = read(handle, remaining)
380 n = len(chunk)
381 if n == 0:
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running the basic_training notebook and inspect RAGTrainer.py at train(), then follow the call into colbert/trainer.py and colbert/infra/launcher.py. Determine why launcher.launch waits on return_value_queue.get() after the training processes start; done means the notebook completes training or reports a clear failure instead of remaining blocked.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.