line / line/lighthouse

AttributeError: `EasyDict` object has no attribute `moment_layers` when loading pre-trained CG-DETR checkpoints.

Open
#74 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
269
Forks
20
Avg merge
7h 3m
Merged PRs (30d)
2

Description

Loading a pre-trained CG-DETR checkpoint from the [uploaded pre-trained weights](https://drive.google.com/file/d/1jxs_bvwttXTF9Lk3aKLohkqfYOonLyrO/view?usp=sharing) fails with the following error:

```
AttributeError: 'EasyDict' object has no attribute 'moment_layers'
```

**To reproduce**
Download pre-trained model weights from the [google drive URL](https://drive.google.com/file/d/1jxs_bvwttXTF9Lk3aKLohkqfYOonLyrO/view?usp=sharing) mentioned in the Readme and then load a CG-DETR checkpoint using `lighthouse.models.CGDETRPredictor`.

```python
from lighthouse.models import CGDETRPredictor
model = CGDETRPredictor('results/cg_detr/qvhighlight/clip/best.ckpt', device='cpu', feature_name='clip')
```

**Full traceback**
```
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Cell In[19], line 1
----> 1 model = CGDETRPredictor('results/cg_detr/qvhighlight/clip/best.ckpt', device='cpu',
2 feature_name='clip')

File .../.venv/lib/python3.9/site-packages/lighthouse/models.py:363, in CGDETRPredictor.__init__(self, ckpt_path, device, feature_name, slowfast_path, pann_path)
355 def __init__(
356 self,
357 ckpt_path: str,
(...)
361 pann_path: Optional[str] = None
362 ) -> None:
--> 363 super().__init__('cg_detr', ckpt_path, device,
364 feature_name, slowfast_path, pann_path)

File .../.venv/lib/python3.9/site-packages/lighthouse/models.py:85, in BasePredictor.__init__(self, model_name, ckpt_path, device, feature_name, slowfast_path, pann_path)
81 self._audio_encoder = self._initialize_audio_encoder(feature_name, pann_path)
83 self._text_encoder: TextEncoder = self._initialize_text_encoder(feature_name)
---> 85 self._model: torch.nn.Module = self._initialize_model(args, model_name)
86 self._load_weights(ckpt['model'])
88 self._feature_name: str = feature_name

File .../.venv/lib/python3.9/site-packages/lighthouse/models.py:107, in BasePredictor._initialize_model(self, args, model_name)
96 model_builders = {
97 'moment_detr': build_model_moment_detr,
98 'qd_detr': build_model_qd_detr,
(...)
103 'taskweave': build_model_task_weave
104 }
106 if model_name in model_builders:
--> 107 model, _ = model_builders[model_name](args)
108 return model
109 else:

File .../.venv/lib/python3.9/site-packages/lighthouse/common/cg_detr.py:998, in build_model(args)
995 def build_model(args):
996 device = torch.device(args.device)
--> 998 transformer = build_transformer(args)
999 position_embedding, txt_position_embedding = build_position_encoding(args)
1001 model = CGDETR(
1002 transformer,
1003 position_embedding,
(...)
1013 args=args
1014 )

File .../.venv/lib/python3.9/site-packages/lighthouse/common/cg_detr_transformer.py:883, in build_transformer(args)
882 def build_transformer(args):
--> 883 return Transformer(
884 d_model=args.hidden_dim,
885 dropout=args.dropout,
886 nhead=args.nheads,
887 dim_feedforward=args.dim_feedforward,
888 num_encoder_layers=args.enc_layers,
889 num_decoder_layers=args.dec_layers,
890 normalize_before=False,
891 return_intermediate_dec=True,
892 activation='prelu',
893 args=args
894 )

File .../.venv/lib/python3.9/site-packages/lighthouse/common/cg_detr_transformer.py:150, in Transformer.__init__(self, d_model, nhead, num_queries, num_encoder_layers, num_decoder_layers, dim_feedforward, dropout, activation, normalize_before, return_intermediate_dec, query_dim, keep_query_pos, query_scale_type, num_patterns, modulate_t_attn, bbox_embed_diff_each_layer, args)
147 mcls_encoder_layer = TransformerEncoderLayer(d_model, nhead, dim_feedforward,
148 dropout, activation, normalize_before)
149 mcls_encoder_norm = nn.LayerNorm(d_model) if normalize_before else None
--> 150 self.mcls_encoder = TransformerEncoder(mcls_encoder_layer, args.moment_layers, mcls_encoder_norm)
152 t2v_encoder_layer = T2V_TransformerEncoderLayer(d_model, nhead, dim_feedforward,
153 dropout, activation, normalize_before, self.args.num_dummies)
154 encoder_norm = nn.LayerNorm(d_model) if normalize_before else None

AttributeError: 'EasyDict' object has no attribute 'moment_layers'
```

The checkpoint doesn't contain the `moment_layers` key although the code expects it to. Maybe the checkpoint was generated on an older version where this attribute wasn't saved?

I attempted to work around this by patching the checkpoint to include `moment_layers`:
```
if not hasattr(ckpt['opt'], 'moment_layers'):
ckpt['opt']['moment_layers'] = 1 # Default value for moment_layers

torch.save(ckpt, 'results/cg_detr/qvhighlight/clip_slowfast/best_patched.ckpt')
```

Loading the model with the patched checkpoint gave a new error: `AttributeError: 'EasyDict' object has no attribute 'num_dummies`.

It seems like these checkpoints are outdated/incompatible with the current Lighthouse version and should either be replaced by new ones or the code made compatible with these checkpoints. I might've just misunderstood the inference instructions though and I'm doing something wrong on my end, in which case sorry for the issue!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing checkpoint loading in lighthouse/models.py, then compare the model construction paths in lighthouse/common/cg_detr.py and lighthouse/common/cg_detr_transformer.py with the saved checkpoint options. Check which configuration keys are required by the current code and absent from the provided weights. Done means the README-linked CG-DETR checkpoint loads through CGDETRPredictor without missing-attribute errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.