isl-org / isl-org/Open3D-ML

Different values for accuracy (validation and testing)

Open
#492 5 comments 7 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.3k
Forks
365
Avg merge
4h 6m
Merged PRs (30d)
1

Description

### Checklist

- [X] I have searched for [similar issues](https://github.com/isl-org/Open3D-ML/issues).
- [X] I have tested with the [latest development wheel](http://www.open3d.org/docs/latest/getting_started.html#development-version-pip).
- [X] I have checked the [release documentation](http://www.open3d.org/docs/release/) and the [latest documentation](http://www.open3d.org/docs/latest/) (for `master` branch).

### Describe the issue

I'm trying to train point transformer on a dataset based on S3DIS.
in the last epoch of the training process, I get a value for accuracy and miou as follows:

INFO - 2022-03-11 10:36:05,257 - semantic_segmentation - Mean acc train: 0.950 eval: 0.933
INFO - 2022-03-11 10:36:05,258 - semantic_segmentation - Mean IoU train: 0.901 eval: 0.902

But when I test my model on the same sample, I get new values for accuracy and miou:
Overall Testing Accuracy : 0.6765742520464817, mIoU : 0.6481393184649503

I'm using pipeline.run_test() to test my model.

I think the value of accuracy is being calculated differently in these stages.

Is there any solution to this problem?

Thank you!

### Steps to reproduce the bug

```python
model = ml3d.models.PointTransformer(**cfg.model)
pipeline = ml3d.pipelines.SemanticSegmentation(model=model, dataset=dataset, device="gpu", **cfg.pipeline)
pipeline.run_train()

# the following lines give the value of accuracy and MIoU in the last epoch:
pipeline.metric_val.acc()
pipeline.metric_val.iou()

# this line of code gives different values of accuracy and MIoU on the same sample of data
pipeline.run_test()
```

### Error message

_No response_

### Expected behavior

the value of acc and MIoU in the last epoch should be equal to the values coming from the function run_test since the same sample for validation and testing has been used.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.