Failed to load lora in int8 mode

Let me introduce the background. The fill model changes the picture background. Torchao quantization is used for inference, but it fails when loading Lora. If Lora is not quantized, it can be loaded.

`transformer = FluxTransformer2DModel.from_pretrained(
model_path
, subfolder = "transformer"
, torch_dtype = torch.bfloat16
)
quantize_device = DEVICE_ID
#int8方式加载
quantize_(
transformer,
int8_weight_only(),
device = quantize_device # quantize using GPU to accelerate the speed
)
#fp8方式加载
# quantize_(
# transformer,
# float8_weight_only(),
# device = quantize_device # quantize using GPU to accelerate the speed
# )
self.pipe = FluxFillPipeline.from_pretrained(
model_path,
transformer = transformer,
torch_dtype = torch.bfloat16
)

if is_add_loramodel:
self.pipe.load_lora_weights("/Flux-Midjourney-Mix2-LoRA/", weight_name="mjV6.safetensors")
self.pipe.fuse_lora(lora_scale=1.2)
#self.pipe.to("cuda:0") #速度快耗费显存全流程不到10s，节省内存 全模型常驻GPU
self.pipe.enable_model_cpu_offload(gpu_id = pipe_gpu_id)#速度慢耗费内存，节省显存空载几乎不消耗显存。`

TypeError: TorchaoLoraLinear.__init__() missing 1 required keyword-only argument: 'get_apply_tensor_subclass'

示されている FluxFillPipeline のセットアップを torchao int8 量子化で再現し、その後 load_lora_weights と fuse_lora を TorchaoLoraLinear.__init__ まで追跡します。量子化された transformer が、get_apply_tensor_subclass 引数が欠けている状態でも LoRA をロードして融合できれば完了です。

索引モデルが issue の本文から書いたものです。