Is it possible to avoid repeated compilation of PromptEncoder.forward?

I'm trying to compile the video predictor by passing vos_optimized = True to build_sam2_video_predictor, but I'm running into issues with the recompilation limit. It seems that for each number of points passed to the PromptEncoder.forward function it needs to get recompiled, until finally pyTorch stops the compilation. Last few lines of output (with TORCH_LOGS="recompiles):

```
V0102 16:04:31.731000 130174120947712 torch/_dynamo/guards.py:2611] [1/8] [__recompiles] Recompiling function forward in /home/steven/SVI/.Python/lib/python3.11/site-packages/sam2/modeling/sam/prompt_encoder.py:160
V0102 16:04:31.731000 130174120947712 torch/_dynamo/guards.py:2611] [1/8] [__recompiles] triggered by the following guard failure(s):
V0102 16:04:31.731000 130174120947712 torch/_dynamo/guards.py:2611] [1/8] [__recompiles] - tensor 'L['points'][0]' stride mismatch at index 0. expected 14, actual 18
V0102 16:04:31.731000 130174120947712 torch/_dynamo/guards.py:2611] [1/8] [__recompiles] - tensor 'L['points'][0]' stride mismatch at index 0. expected 12, actual 18
V0102 16:04:31.731000 130174120947712 torch/_dynamo/guards.py:2611] [1/8] [__recompiles] - tensor 'L['points'][0]' stride mismatch at index 0. expected 16, actual 18
V0102 16:04:31.731000 130174120947712 torch/_dynamo/guards.py:2611] [1/8] [__recompiles] - tensor 'L['points'][0]' stride mismatch at index 0. expected 10, actual 18
V0102 16:04:31.731000 130174120947712 torch/_dynamo/guards.py:2611] [1/8] [__recompiles] - tensor 'L['points'][0]' stride mismatch at index 0. expected 8, actual 18
V0102 16:04:31.731000 130174120947712 torch/_dynamo/guards.py:2611] [1/8] [__recompiles] - tensor 'L['points'][0]' stride mismatch at index 0. expected 6, actual 18
V0102 16:04:31.731000 130174120947712 torch/_dynamo/guards.py:2611] [1/8] [__recompiles] - tensor 'L['points'][0]' stride mismatch at index 0. expected 2, actual 18
V0102 16:04:31.731000 130174120947712 torch/_dynamo/guards.py:2611] [1/8] [__recompiles] - expected type of 'L['masks']' to be a tensor type, ' but found
W0102 16:04:31.731000 130174120947712 torch/_dynamo/convert_frame.py:762] [1/8] torch._dynamo hit config.cache_size_limit (8)
W0102 16:04:31.731000 130174120947712 torch/_dynamo/convert_frame.py:762] [1/8] function: 'forward' (/home/steven/SVI/.Python/lib/python3.11/site-packages/sam2/modeling/sam/prompt_encoder.py:160)
W0102 16:04:31.731000 130174120947712 torch/_dynamo/convert_frame.py:762] [1/8] last reason: expected type of 'L['masks']' to be a tensor type, ' but found
W0102 16:04:31.731000 130174120947712 torch/_dynamo/convert_frame.py:762] [1/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
W0102 16:04:31.731000 130174120947712 torch/_dynamo/convert_frame.py:762] [1/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html.
```

MaskDecoder.forward seems to be nearing the limit as well:

```
V0102 16:04:18.120000 130174120947712 torch/_dynamo/guards.py:2611] [2/7] [__recompiles] Recompiling function forward in /home/steven/SVI/.Python/lib/python3.11/site-packages/sam2/modeling/sam/mask_decoder.py:110
V0102 16:04:18.120000 130174120947712 torch/_dynamo/guards.py:2611] [2/7] [__recompiles] triggered by the following guard failure(s):
V0102 16:04:18.120000 130174120947712 torch/_dynamo/guards.py:2611] [2/7] [__recompiles] - tensor 'L['sparse_prompt_embeddings']' stride mismatch at index 0. expected 1536, actual 2304
V0102 16:04:18.120000 130174120947712 torch/_dynamo/guards.py:2611] [2/7] [__recompiles] - tensor 'L['sparse_prompt_embeddings']' stride mismatch at index 0. expected 1792, actual 2304
V0102 16:04:18.120000 130174120947712 torch/_dynamo/guards.py:2611] [2/7] [__recompiles] - tensor 'L['sparse_prompt_embeddings']' stride mismatch at index 0. expected 2048, actual 2304
V0102 16:04:18.120000 130174120947712 torch/_dynamo/guards.py:2611] [2/7] [__recompiles] - tensor 'L['sparse_prompt_embeddings']' stride mismatch at index 0. expected 1280, actual 2304
V0102 16:04:18.120000 130174120947712 torch/_dynamo/guards.py:2611] [2/7] [__recompiles] - tensor 'L['sparse_prompt_embeddings']' stride mismatch at index 0. expected 1024, actual 2304
V0102 16:04:18.120000 130174120947712 torch/_dynamo/guards.py:2611] [2/7] [__recompiles] - ___check_obj_id(L['multimask_output'], 36586624)
V0102 16:04:18.120000 130174120947712 torch/_dynamo/guards.py:2611] [2/7] [__recompiles] - tensor 'L['dense_prompt_embeddings']' dtype mismatch. expected Half, actual Float
```

I can probably avoid the error by increasing the cache limit, but is it also possible to avoid the recompilations altogether?