isl-org / isl-org/Open3D-ML

Missing Keys and Unexpected Keys occured while training RandLANet on SemanticKITTI

Open
#483 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.3k
Forks
365
Avg merge
4h 6m
Merged PRs (30d)
1

Description

### Checklist

- [X] I have searched for [similar issues](https://github.com/isl-org/Open3D-ML/issues).
- [X] I have tested with the [latest development wheel](http://www.open3d.org/docs/latest/getting_started.html#development-version-pip).
- [X] I have checked the [release documentation](http://www.open3d.org/docs/release/) and the [latest documentation](http://www.open3d.org/docs/latest/) (for `master` branch).

### Describe the issue

I have followed simple instructions from **Semantic Segmentation** from `README.md`. But the link randlanet_url seems to be invalid and there was **403 Forbidden** from downloaded file. So I resorted to the URL given by **Model Zoo** section, where I was able to download file `randlanet_semantickitti_202201071330utc.pth`.

However, when I tried to load parameters from the pretrained model, error occured in loading state_dict for RandLANet. I guess the pretrained model is generated using different configuration of RandLANet, so I am here to consult which modifications should I make to load pretrained parameters correctly?

### Steps to reproduce the bug

```python
import os
import open3d.ml as _ml3d
import open3d.ml.torch as ml3d

# Running a pretrained model for semantic segmentation
cfg_file = "../ml3d/configs/randlanet_semantickitti.yml"
cfg = _ml3d.utils.Config.load_from_file(cfg_file)

model = ml3d.models.RandLANet(**cfg.model)
cfg.dataset['dataset_path'] = "/home/jiayelin/Datasets/data_odometry_velodyne"
dataset = ml3d.datasets.SemanticKITTI(cfg.dataset.pop('dataset_path', None), **cfg.dataset)
pipeline = ml3d.pipelines.SemanticSegmentation(model, dataset=dataset, device="gpu", **cfg.pipeline)

# download the weights
ckpt_folder = "../logs/"
os.makedirs(ckpt_folder, exist_ok=True)
# I modified path and url to model provided by Model Zoo
ckpt_path = ckpt_folder + "randlanet_semantickitti_202201071330utc.pth"
randlanet_url = "https://storage.googleapis.com/open3d-releases/model-zoo/randlanet_semantickitti_202201071330utc.pth"
if not os.path.exists(ckpt_path):
cmd = "wget {} -o {}".format(randlanet_url, ckpt_path)
os.system(cmd)

# load the parameters.
pipeline.load_ckpt(ckpt_path=ckpt_path) # where error occured

test_split = dataset.get_split("test")
data = test_split.get_data(0)

# run inference on a single example
# returns dict with 'predict_labels' and 'predict_scores'
result = pipeline.run_inference(data)

# evaluate performance on the test set; this will write logs to '../logs'
pipeline.run_test()
```

### Error message

2022-03-01 15:26:46.858921: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
INFO - 2022-03-01 15:26:47,672 - semantic_segmentation - Loading checkpoint ../logs/randlanet_semantickitti_202201071330utc.pth
Traceback (most recent call last):
File "/home/jiayelin/Projects/Open3D-ML/examples/semantic_segmentation.py", line 24, in
pipeline.load_ckpt(ckpt_path=ckpt_path)
File "/home/jiayelin/.local/lib/python3.8/site-packages/open3d/_ml3d/torch/pipelines/semantic_segmentation.py", line 667, in load_ckpt
self.model.load_state_dict(ckpt['model_state_dict'])
File "/home/jiayelin/Software/anaconda3/envs/tf2_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1223, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for RandLANet:
Missing key(s) in state_dict: "batch_normalization.weight", "batch_normalization.bias", "batch_normalization.running_mean", "batch_normalization.running_var", "Encoder_layer_0mlp1.biases", "Encoder_layer_0mlp1.weights" ......
Unexpected key(s) in state_dict: "bn0.weight", "bn0.bias", "bn0.running_mean", "bn0.running_var", "bn0.num_batches_tracked", "encoder.0.mlp1.conv.weight", "encoder.0.mlp1.conv.bias", "encoder.0.mlp1.batch_norm.weight", "encoder.0.mlp1.batch_norm.bias" .......

### Expected behavior

Parameters from pretrained model should be loaded successfully and inference process is guaranteed.

### Open3D, Python and System information

```markdown
- Operating system: Ubuntu 20.04
- Python version: Python 3.8.12
- Open3D version: 0.14.1
- System type: x84
- Is this remote workstation?: no
- How did you install Open3D?: pip
- Compiler version (if built from source):
```

### Additional information

_No response_

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.