AttributeError: 'DaViT' object has no attribute '_initialize_weights'

Florence2LanguageForConditionalGeneration has generative capabilities, as prepare_inputs_for_generation is explicitly defined. However, it doesn't directly inherit from GenerationMixin. From 👉v4.50👈 onwards, PreTrainedModel will NOT inherit from GenerationMixin, and this model will lose the ability to call generate and other related functions.

• If you're using trust_remote_code=True, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
• If you are the owner of the model architecture code, please modify your model class such that it inherits from GenerationMixin (after PreTrainedModel, otherwise you'll get an exception).
• If you are not the owner of the model architecture class, please contact the model code owner to update it.
  Traceback (most recent call last):
  File "c:\Users\LENOVO\Dropbox\My PC (DESKTOP-RQ2BTG4)\Desktop\test\src\main.py", line 6, in
  from screens.screenshot_collector import screenshot_collector_worker
  File "c:\Users\LENOVO\Dropbox\My PC (DESKTOP-RQ2BTG4)\Desktop\test\src\screens\screenshot_collector.py", line 33, in
  caption_model_processor = get_caption_model_processor(
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\LENOVO\Dropbox\My PC (DESKTOP-RQ2BTG4)\Desktop\test\src\OmniParser\util\utils.py", line 326, in get_caption_model_processor
  model = AutoModelForCausalLM.from_pretrained(model_name_or_path, torch_dtype=torch.float16, trust_remote_code=True).to(device)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\LENOVO\miniconda3\envs\omni\Lib\site-packages\transformers\models\auto\auto_factory.py", line 564, in from_pretrained
  return model_class.from_pretrained(
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\LENOVO\miniconda3\envs\omni\Lib\site-packages\transformers\modeling_utils.py", line 309, in _wrapper
  return func(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\LENOVO\miniconda3\envs\omni\Lib\site-packages\transformers\modeling_utils.py", line 4574, in from_pretrained
  ) = cls._load_pretrained_model(
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\LENOVO\miniconda3\envs\omni\Lib\site-packages\transformers\modeling_utils.py", line 4884, in _load_pretrained_model
  model._initialize_missing_keys(checkpoint_keys, ignore_mismatched_sizes, is_quantized)
  File "C:\Users\LENOVO\miniconda3\envs\omni\Lib\site-packages\transformers\modeling_utils.py", line 5477, in _initialize_missing_keys
  self.initialize_weights()
  File "C:\Users\LENOVO\miniconda3\envs\omni\Lib\site-packages\torch\utils_contextlib.py", line 116, in decorate_context
  return func(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\LENOVO\miniconda3\envs\omni\Lib\site-packages\transformers\modeling_utils.py", line 2556, in initialize_weights
  self.smart_apply(self._initialize_weights)
  File "C:\Users\LENOVO\miniconda3\envs\omni\Lib\site-packages\transformers\modeling_utils.py", line 2547, in smart_apply
  module.smart_apply(module._initialize_weights)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\LENOVO\miniconda3\envs\omni\Lib\site-packages\torch\nn\modules\module.py", line 1940, in getattr
  raise AttributeError(
  AttributeError: 'DaViT' object has no attribute '_initialize_weights'

Hello, i am a very new developper and was wondering what could cause this issue if anyone could help. Thanks!

Start at src/OmniParser/util/utils.py around line 326, where AutoModelForCausalLM loads the caption model, and trace the call from src/screens/screenshot_collector.py line 33. Reproduce the failure with the reported model and inspect the DaViT class and installed Transformers version. Done means the caption model loads without the _initialize_weights AttributeError and the existing screenshot collector startup still works.