Dynamic shape support needed for a number of MoE models through ThunderFX

Nobody has claimed this yet.

## 🚀 Feature

This issue presents a new scenario that requires dynamic shape support in ThunderFX.

### Motivation

For HF `allenai/OLMoE-1B-7B-0924`, `microsoft/GRIN-MoE`, `microsoft/Phi-3.5-MoE-instruct`, `Undi95/dbrx-base`, the benchmark runs very slow:
```
python benchmark_peft.py --model Undi95/dbrx-base --trust-remote-code --attn-implementation eager --max-steps 100 --mbs 1 --seq-length 4096 --compile thunder --fixed-num-hidden-layers 2
python benchmark_peft.py --model microsoft/GRIN-MoE --trust-remote-code --attn-implementation sdpa --max-steps 100 --mbs 1 --seq-length 4096 --compile thunder --fixed-num-hidden-layers 2
python benchmark_peft.py --model microsoft/Phi-3.5-MoE-instruct --trust-remote-code --attn-implementation eager --max-steps 100 --mbs 1 --seq-length 4096 --compile thunder --fixed-num-hidden-layers 2
python benchmark_peft.py --model allenai/OLMoE-1B-7B-0924 --trust-remote-code --attn-implementation sdpa --max-steps 100 --mbs 1 --seq-length 4096 --compile thunder --fixed-num-hidden-layers 2
```
The reason is that Dynamo segments the model into multiple subgraphs, and one of these subgraphs receives dynamic input, which causes Thunder to recompile each time.
An example subgraph:
```
class DynamoModule(torch.nn.Module):
def forward(self, l_x_: "bf16[s77, 2048][2048, 1]cuda:0", s77: "Sym(s77)", l_self_modules_gate_proj_modules_base_layer_parameters_weight_: "bf16[1024, 2048][2048, 1]cuda:0", l_self_modules_gate_proj_modules_lora_a_modules_default_parameters_weight_: "f32[16, 2048][2048, 1]cuda:0", l_self_modules_gate_proj_modules_lora_b_modules_default_parameters_weight_: "f32[1024, 16][16, 1]cuda:0", l_self_modules_up_proj_modules_base_layer_parameters_weight_: "bf16[1024, 2048][2048, 1]cuda:0", l_self_modules_up_proj_modules_lora_a_modules_default_parameters_weight_: "f32[16, 2048][2048, 1]cuda:0", l_self_modules_up_proj_modules_lora_b_modules_default_parameters_weight_: "f32[1024, 16][16, 1]cuda:0", l_self_modules_down_proj_modules_base_layer_parameters_weight_: "bf16[2048, 1024][1024, 1]cuda:0", l_self_modules_down_proj_modules_lora_a_modules_default_parameters_weight_: "f32[16, 1024][1024, 1]cuda:0", l_self_modules_down_proj_modules_lora_b_modules_default_parameters_weight_: "f32[2048, 16][16, 1]cuda:0"):
# File: /usr/local/lib/python3.12/dist-packages/peft/tuners/lora/layer.py:755 in forward, code: result = self.base_layer(x, *args, **kwargs)
result: "bf16[s77, 1024][1024, 1]cuda:0" = torch._C._nn.linear(l_x_, l_self_modules_gate_proj_modules_base_layer_parameters_weight_, None); l_self_modules_gate_proj_modules_base_layer_parameters_weight_ = None

# File: /usr/local/lib/python3.12/dist-packages/peft/tuners/tuners_utils.py:931 in _cast_input_dtype, code: return x.to(dtype=dtype)
x: "f32[s77, 2048][2048, 1]cuda:0" = l_x_.to(dtype = torch.float32)

# File: /usr/local/lib/python3.12/dist-packages/peft/tuners/lora/layer.py:769 in forward, code: result = result + lora_B(lora_A(dropout(x))) * scaling
linear_1: "f32[s77, 16][16, 1]cuda:0" = torch._C._nn.linear(x, l_self_modules_gate_proj_modules_lora_a_modules_default_parameters_weight_, None); x = l_self_modules_gate_proj_modules_lora_a_modules_default_parameters_weight_ = None
linear_2: "f32[s77, 1024][1024, 1]cuda:0" = torch._C._nn.linear(linear_1, l_self_modules_gate_proj_modules_lora_b_modules_default_parameters_weight_, None); linear_1 = l_self_modules_gate_proj_modules_lora_b_modules_default_parameters_weight_ = None
mul: "f32[s77, 1024][1024, 1]cuda:0" = linear_2 * 2.0; linear_2 = None
result_1: "f32[s77, 1024][1024, 1]cuda:0" = result + mul; result = mul = None

# File: /usr/local/lib/python3.12/dist-packages/peft/tuners/lora/layer.py:778 in forward, code: result = result.to(torch_result_dtype)
result_2: "bf16[s77, 1024][1024, 1]cuda:0" = result_1.to(torch.bfloat16); result_1 = None

# File: /usr/local/lib/python3.12/dist-packages/transformers/models/olmoe/modeling_olmoe.py:232 in forward, code: down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
silu: "bf16[s77, 1024][1024, 1]cuda:0" = torch.nn.functional.silu(result_2, inplace = False); result_2 = None

# File: /usr/local/lib/python3.12/dist-packages/peft/tuners/lora/layer.py:755 in forward, code: result = self.base_layer(x, *args, **kwargs)
result_3: "bf16[s77, 1024][1024, 1]cuda:0" = torch._C._nn.linear(l_x_, l_self_modules_up_proj_modules_base_layer_parameters_weight_, None); l_self_modules_up_proj_modules_base_layer_parameters_weight_ = None

# File: /usr/local/lib/python3.12/dist-packages/peft/tuners/tuners_utils.py:931 in _cast_input_dtype, code: return x.to(dtype=dtype)
x_1: "f32[s77, 2048][2048, 1]cuda:0" = l_x_.to(dtype = torch.float32); l_x_ = None

# File: /usr/local/lib/python3.12/dist-packages/peft/tuners/lora/layer.py:769 in forward, code: result = result + lora_B(lora_A(dropout(x))) * scaling
linear_4: "f32[s77, 16][16, 1]cuda:0" = torch._C._nn.linear(x_1, l_self_modules_up_proj_modules_lora_a_modules_default_parameters_weight_, None); x_1 = l_self_modules_up_proj_modules_lora_a_modules_default_parameters_weight_ = None
linear_5: "f32[s77, 1024][1024, 1]cuda:0" = torch._C._nn.linear(linear_4, l_self_modules_up_proj_modules_lora_b_modules_default_parameters_weight_, None); linear_4 = l_self_modules_up_proj_modules_lora_b_modules_default_parameters_weight_ = None
mul_1: "f32[s77, 1024][1024, 1]cuda:0" = linear_5 * 2.0; linear_5 = None
result_4: "f32[s77, 1024][1024, 1]cuda:0" = result_3 + mul_1; result_3 = mul_1 = None

# File: /usr/local/lib/python3.12/dist-packages/peft/tuners/lora/layer.py:778 in forward, code: result = result.to(torch_result_dtype)
result_5: "bf16[s77, 1024][1024, 1]cuda:0" = result_4.to(torch.bfloat16); result_4 = None

# File: /usr/local/lib/python3.12/dist-packages/transformers/models/olmoe/modeling_olmoe.py:232 in forward, code: down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
mul_2: "bf16[s77, 1024][1024, 1]cuda:0" = silu * result_5; silu = result_5 = None

# File: /usr/local/lib/python3.12/dist-packages/peft/tuners/lora/layer.py:755 in forward, code: result = self.base_layer(x, *args, **kwargs)
result_6: "bf16[s77, 2048][2048, 1]cuda:0" = torch._C._nn.linear(mul_2, l_self_modules_down_proj_modules_base_layer_parameters_weight_, None); l_self_modules_down_proj_modules_base_layer_parameters_weight_ = None

# File: /usr/local/lib/python3.12/dist-packages/peft/tuners/tuners_utils.py:931 in _cast_input_dtype, code: return x.to(dtype=dtype)
x_2: "f32[s77, 1024][1024, 1]cuda:0" = mul_2.to(dtype = torch.float32); mul_2 = None

# File: /usr/local/lib/python3.12/dist-packages/peft/tuners/lora/layer.py:769 in forward, code: result = result + lora_B(lora_A(dropout(x))) * scaling
linear_7: "f32[s77, 16][16, 1]cuda:0" = torch._C._nn.linear(x_2, l_self_modules_down_proj_modules_lora_a_modules_default_parameters_weight_, None); x_2 = l_self_modules_down_proj_modules_lora_a_modules_default_parameters_weight_ = None
linear_8: "f32[s77, 2048][2048, 1]cuda:0" = torch._C._nn.linear(linear_7, l_self_modules_down_proj_modules_lora_b_modules_default_parameters_weight_, None); linear_7 = l_self_modules_down_proj_modules_lora_b_modules_default_parameters_weight_ = None
mul_3: "f32[s77, 2048][2048, 1]cuda:0" = linear_8 * 2.0; linear_8 = None
result_7: "f32[s77, 2048][2048, 1]cuda:0" = result_6 + mul_3; result_6 = mul_3 = None

# File: /usr/local/lib/python3.12/dist-packages/peft/tuners/lora/layer.py:778 in forward, code: result = result.to(torch_result_dtype)
result_8: "bf16[s77, 2048][2048, 1]cuda:0" = result_7.to(torch.bfloat16); result_7 = None
return result_8
```

Start with benchmark_peft.py and reproduce one of the listed MoE model commands, then inspect the Dynamo subgraph shown in the issue and its dynamic input shape. Done means ThunderFX supports these dynamic shapes without recompiling the affected subgraph on each run and the benchmark no longer suffers the reported slowdown.

Written by the indexing model from the issue text.