Memory management sucks after 0.30v

The pr #15027 is probably messing up with RAM management, at least for me. I can still remember, qwen3-vl-4b was only 3-4gb fully loaded, then the 0.30 update destroyed this completely. Now it uses almost my entire Raw to keep unnecessary weights (no, I got no speed up or benefits with this implementation). I don't know how much this works, but it doesn't feel good have my RAM filled with wtf stuff. Sorry for unfortunate words (and the title as well), but it is what I feel now.

The issue names no files or tests; begin by reviewing PR #15027 and the memory-management changes introduced in the 0.30 update. Reproduce the report with qwen3-vl-4b and compare RAM usage before and after the update. Done means determining whether the regression is caused by that change and recording measured memory behavior.

Written by the indexing model from the issue text.