[Bug]: auto-scheme wrongly changes dataset when oom
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 175
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 99
Description
### Problem Description
Generating AutoScheme: 9%|████████████████████ | 383/4096 [03:52<1:15:07, 1.21s/it][W818 19:47:57.206225105 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 8136949760 bytes (free: 7246446592, total: 85094825984).
[W818 19:47:57.305789898 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 8136949760 bytes (free: 7252738048, total: 85094825984).
2026-08-18 19:47:57 WARNING delta_loss.py L1133: Text-only calibration failed on VLM (CUDA out of memory. Tried to allocate 7.58 GiB. GPU 0 has a total capacity of 79.25 GiB of which 6.75 GiB is free. Process 230003 has 6.11 GiB memory in use. Process 230359 has 33.18 GiB memory in use. Including non-PyTorch memory, this process has 33.19 GiB memory in use. Of the allocated memory 24.51 GiB is allocated by PyTorch, and 8.13 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)); falling back to multimodal calibration dataloader.
Generating AutoScheme: 9%|████████████████████▏ | 384/4096 [03:53<1:14:02, 1.20s/it]2026-08-18 19:47:57 WARNING logging.py L340: AutoScheme(force_mllm): dataset='NeelNanda/pile-10k' is text-only, overriding to 'liuhaotian/llava_conv_58k'.
2026-08-18 19:47:57 WARNING logging.py L340: Unable to recognize qwen3_5, using default template instead.
2026-08-18 19:47:57 WARNING dataset.py L250: seqlen(1024) is greater than the maximum length supported by the liuhaotian/llava_conv_58k, reset to 512
### Reproduction Steps
~
### Environment Information
_No response_
### Error Logs
```shell
```
### Additional Context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.