Train the SAM directly

I tried to train the sam model with checkpoint directly, but it had no change after loss.backward() though I use sam.train() before. Do I need other configuration?

The issue names no file or test. Start by reproducing the checkpoint-based training attempt with sam.train() and loss.backward(), then inspect how the checkpoint and optimizer configuration are connected. Done means identifying the required configuration or confirming a documented limitation with a minimal reproduction and explanation.

Written by the indexing model from the issue text.