Alberto-Codes / Alberto-Codes/vramfit

Does a fine-tune inherit its base checkpoint's sensitivity map?

Open
#412 1 comment 0 reactions 0 assignees View on GitHub
question research
Dominant language
Python
Stars
1
Forks
1
Avg merge
6h 8m
Merged PRs (30d)
105

Description

## Question

Does a fine-tune inherit its base checkpoint's sensitivity map? If the
ranking survives through every rank the solver reads, then one base
scan serves every fine-tune of that model.

## Why it is live

The fine-tune-fitting lane parked on #68 prices each job at one scan.
A scan is the expensive step. If a LoRA or QLoRA fine-tune leaves the
damage ordering intact, the lane's unit cost collapses to a plan and a
pack. If it does not, every fine-tune pays a full scan, and the lane
carries a different economics.

Nothing in the records answers this. Every map measured so far reads
one checkpoint.

## The experiment already has a shape

Publication #2 ran the neighbouring test. Its card records:

> "A second sensitivity map, measured without the importance matrix,
> agrees with this ordering through every rank the solve reads and
> yields the identical placement."

That is map stability across a measurement change. This ticket asks for
map stability across a weight change. The comparison method carries
over: scan both checkpoints, compare the ordering through the ranks the
solve reads, and check whether the placement lands identical.

## What a result would settle

1. Whether a fine-tune needs its own scan, or reads the base map.
2. Whether the answer depends on the adapter rank, the epoch count, or
the fraction of weights the merge moves. A rank-16 QLoRA over 3
epochs moves less than a full fine-tune.
3. Whether a damage ordering is a property of the architecture or of
the trained weights. That reaches further than this lane.

## Cost

Two scans of one small dense target, plus one merged fine-tune of it.
Pick a target small enough that a scan is cheap, and dense, so the
expert-stack machinery stays out of the frame.

## Open

Which target? A small dense model keeps the scan cheap. The
answer may not generalize from small to large, and the ticket should
say so when it closes.

Does a merged QLoRA adapter count as a fine-tune for this purpose, or
does the question need a full fine-tune to be worth answering?

## Origin

Raised 2026-08-22 from an r/LocalLLaMA post recorded on #68. The author
fine-tuned Gemma 4 12B under a 16 GiB constraint. Their case is the
lane's shape, and their checkpoint would need a scan today.

## Related

- #68 parks the lane this prices.
- #144 asks whether per-layer sensitivity survives expert routing. This
asks whether it survives a weight update.

Contributor guide

Open the contributing guide

Research direction

The payload names no files, tests, or entry points. Start by selecting a small dense target and run the stated comparison: scan the base and merged fine-tune, compare ordering through solver-read ranks, and check identical placement. Record adapter rank, epoch count, and moved-weight fraction, then document whether a separate scan is needed and whether results generalize.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.