microsoft / microsoft/KBLaM

After running train nothing is happening. Stuck on Running on Cuda.

Open
#41 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
1.5k
Forks
125
Avg merge
22d 8m
Merged PRs (30d)
1

Description

Hello,

I'm trying to recreate the experiments and after running train nothing is happening:

!python experiments/train.py --dataset_dir ../datasets --train_dataset synthetic --N 120000 --B 20 --total_steps 601 --encoder_spec ALLMINILM --key_embd_src key --use_data_aug --log_to_file --verbose

Output:

wandb: Using wandb-core as the SDK backend. Please refer to https://wandb.me/wandb-core for more information.
wandb: (1) Create a W&B account
wandb: (2) Use an existing W&B account
wandb: (3) Don't visualize my results
wandb: Enter your choice: 3
wandb: You chose "Don't visualize my results"
wandb: Tracking run with wandb version 0.19.9
wandb: W&B syncing is set to offline in this directory.
wandb: Run wandb online or set WANDB_MODE=online to enable cloud syncing.
[10:01:24] INFO Running on cuda
---- Just stops here ---

The log file shows:

2025-04-10 10:00:08,854 - training - INFO - Running on cuda
2025-04-10 10:00:08,857 - training - INFO - 🚨 Started training 🚨
2025-04-10 10:00:08,859 - training - INFO - 💽 Saving to output💽
2025-04-10 10:00:08,860 - training - INFO - Using pre-computed ALLMINILM embedding
--- Stops here ---

Using Google Colab with T4.

Created embeddings using all-MiniLM-L6-v2.

Could you help me with this ?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the command in experiments/train.py on the reported Colab T4 setup, then trace the execution after the “Using pre-computed ALLMINILM embedding” log entry. Check whether training advances or blocks at that point and document the cause. Done means training starts progressing or the failure has a clear, reproducible diagnosis.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.