Missing Keys and Unexpected Keys occured while training RandLANet on SemanticKITTI
- Dominant language
- Python
- Stars
- 2.3k
- Forks
- 365
- Avg merge
- 4h 6m
- Merged PRs (30d)
- 1
Description
### Checklist
- [X] I have searched for [similar issues](https://github.com/isl-org/Open3D-ML/issues).
- [X] I have tested with the [latest development wheel](http://www.open3d.org/docs/latest/getting_started.html#development-version-pip).
- [X] I have checked the [release documentation](http://www.open3d.org/docs/release/) and the [latest documentation](http://www.open3d.org/docs/latest/) (for `master` branch).
### Describe the issue
I have followed simple instructions from **Semantic Segmentation** from `README.md`. But the link randlanet_url seems to be invalid and there was **403 Forbidden** from downloaded file. So I resorted to the URL given by **Model Zoo** section, where I was able to download file `randlanet_semantickitti_202201071330utc.pth`.
However, when I tried to load parameters from the pretrained model, error occured in loading state_dict for RandLANet. I guess the pretrained model is generated using different configuration of RandLANet, so I am here to consult which modifications should I make to load pretrained parameters correctly?
### Steps to reproduce the bug
```python
import os
import open3d.ml as _ml3d
import open3d.ml.torch as ml3d
# Running a pretrained model for semantic segmentation
cfg_file = "../ml3d/configs/randlanet_semantickitti.yml"
cfg = _ml3d.utils.Config.load_from_file(cfg_file)
model = ml3d.models.RandLANet(**cfg.model)
cfg.dataset['dataset_path'] = "/home/jiayelin/Datasets/data_odometry_velodyne"
dataset = ml3d.datasets.SemanticKITTI(cfg.dataset.pop('dataset_path', None), **cfg.dataset)
pipeline = ml3d.pipelines.SemanticSegmentation(model, dataset=dataset, device="gpu", **cfg.pipeline)
# download the weights
ckpt_folder = "../logs/"
os.makedirs(ckpt_folder, exist_ok=True)
# I modified path and url to model provided by Model Zoo
ckpt_path = ckpt_folder + "randlanet_semantickitti_202201071330utc.pth"
randlanet_url = "https://storage.googleapis.com/open3d-releases/model-zoo/randlanet_semantickitti_202201071330utc.pth"
if not os.path.exists(ckpt_path):
cmd = "wget {} -o {}".format(randlanet_url, ckpt_path)
os.system(cmd)
# load the parameters.
pipeline.load_ckpt(ckpt_path=ckpt_path) # where error occured
test_split = dataset.get_split("test")
data = test_split.get_data(0)
# run inference on a single example
# returns dict with 'predict_labels' and 'predict_scores'
result = pipeline.run_inference(data)
# evaluate performance on the test set; this will write logs to '../logs'
pipeline.run_test()
```
### Error message
2022-03-01 15:26:46.858921: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
INFO - 2022-03-01 15:26:47,672 - semantic_segmentation - Loading checkpoint ../logs/randlanet_semantickitti_202201071330utc.pth
Traceback (most recent call last):
File "/home/jiayelin/Projects/Open3D-ML/examples/semantic_segmentation.py", line 24, in
pipeline.load_ckpt(ckpt_path=ckpt_path)
File "/home/jiayelin/.local/lib/python3.8/site-packages/open3d/_ml3d/torch/pipelines/semantic_segmentation.py", line 667, in load_ckpt
self.model.load_state_dict(ckpt['model_state_dict'])
File "/home/jiayelin/Software/anaconda3/envs/tf2_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1223, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for RandLANet:
Missing key(s) in state_dict: "batch_normalization.weight", "batch_normalization.bias", "batch_normalization.running_mean", "batch_normalization.running_var", "Encoder_layer_0mlp1.biases", "Encoder_layer_0mlp1.weights" ......
Unexpected key(s) in state_dict: "bn0.weight", "bn0.bias", "bn0.running_mean", "bn0.running_var", "bn0.num_batches_tracked", "encoder.0.mlp1.conv.weight", "encoder.0.mlp1.conv.bias", "encoder.0.mlp1.batch_norm.weight", "encoder.0.mlp1.batch_norm.bias" .......
### Expected behavior
Parameters from pretrained model should be loaded successfully and inference process is guaranteed.
### Open3D, Python and System information
```markdown
- Operating system: Ubuntu 20.04
- Python version: Python 3.8.12
- Open3D version: 0.14.1
- System type: x84
- Is this remote workstation?: no
- How did you install Open3D?: pip
- Compiler version (if built from source):
```
### Additional information
_No response_
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.