OpenPipe / OpenPipe/ART

Cuda error in RULER notebook

Open
#436 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
10.8k
Forks
989
Avg merge
6h 29m
Merged PRs (30d)
85

Description

I am using the RULER notebook to train a model, but I get this error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/usr/local/lib/python3.12/dist-packages/unsloth_zoo/vllm_utils.py in load_vllm(model_name, config, gpu_memory_utilization, max_seq_length, dtype, training, float8_kv_cache, random_state, enable_lora, max_lora_rank, max_loras, use_async, use_engine, disable_log_stats, enforce_eager, enable_prefix_caching, compilation_config, conservativeness, max_logprobs, use_bitsandbytes, unsloth_vllm_standby, return_args)
   1499             if use_async:
-> 1500                 llm = AsyncLLMEngine.from_engine_args(AsyncEngineArgs(**engine_args))
   1501             elif use_engine:

31 frames
RuntimeError: torch.cuda.MemPool doesn't currently support expandable_segments.

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
/usr/local/lib/python3.12/dist-packages/unsloth_zoo/vllm_utils.py in load_vllm(model_name, config, gpu_memory_utilization, max_seq_length, dtype, training, float8_kv_cache, random_state, enable_lora, max_lora_rank, max_loras, use_async, use_engine, disable_log_stats, enforce_eager, enable_prefix_caching, compilation_config, conservativeness, max_logprobs, use_bitsandbytes, unsloth_vllm_standby, return_args)
   1525                 )
   1526             else:
-> 1527                 raise RuntimeError(error)
   1528         pass
   1529     pass

RuntimeError: torch.cuda.MemPool doesn't currently support expandable_segments.

I have tried upgrading transformers, vllm, and ART, and I have also tried multiple models, including GPT-OSS 20b and Qwen/Qwen2.5-7B-Instruct, but nothing resolved this issue. Here is my notebook's code: https://colab.research.google.com/drive/13Ax7eQ313WxTHXzosUciHdBYXlnG9047?usp=sharing

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Open the linked Google Colab notebook and first run the RULER training flow that reaches vLLM, recording the environment and the reported torch.cuda.MemPool error. Done means the notebook can load a selected model and start training without this runtime error, with the result documented in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.