Failed to load lora in int8 mode

Let me introduce the background. The fill model changes the picture background. Torchao quantization is used for inference, but it fails when loading Lora. If Lora is not quantized, it can be loaded.

`transformer = FluxTransformer2DModel.from_pretrained(
model_path
, subfolder = "transformer"
, torch_dtype = torch.bfloat16
)
quantize_device = DEVICE_ID
#int8方式加载
quantize_(
transformer,
int8_weight_only(),
device = quantize_device # quantize using GPU to accelerate the speed
)
#fp8方式加载
# quantize_(
# transformer,
# float8_weight_only(),
# device = quantize_device # quantize using GPU to accelerate the speed
# )
self.pipe = FluxFillPipeline.from_pretrained(
model_path,
transformer = transformer,
torch_dtype = torch.bfloat16
)

if is_add_loramodel:
self.pipe.load_lora_weights("/Flux-Midjourney-Mix2-LoRA/", weight_name="mjV6.safetensors")
self.pipe.fuse_lora(lora_scale=1.2)
#self.pipe.to("cuda:0") #速度快耗费显存全流程不到10s，节省内存 全模型常驻GPU
self.pipe.enable_model_cpu_offload(gpu_id = pipe_gpu_id)#速度慢耗费内存，节省显存空载几乎不消耗显存。`

TypeError: TorchaoLoraLinear.__init__() missing 1 required keyword-only argument: 'get_apply_tensor_subclass'

Reproduce the shown FluxFillPipeline setup with torchao int8 quantization, then trace load_lora_weights and fuse_lora into TorchaoLoraLinear.__init__. Done means the quantized transformer can load and fuse the LoRA without the missing get_apply_tensor_subclass argument.

Escrito por el modelo de indexación a partir del texto del issue.