facebookresearch / facebookresearch/easy3d

Performance gap when training on ScanNet and testing on KITTI-360

Open
#3 3 comments 0 reactions 1 assignee Claimed by @simo23 View on GitHub
Dominant language
Python
Stars
28
Forks
3
PR merge metrics
No merged PRs in 30d

Description

### Description
Hello, thank you for the excellent work on this project and the accompanying paper.

I am writing to report a significant performance gap that I observed when reproducing cross-dataset evaluation results, and I would like to ask for clarification on whether this behavior is expected.

### Experimental Setup

- Training dataset: /data/agile3d/ScanNet

- Testing dataset: /data/agile3d/KITTI360

- Training procedure:

- Followed the official training pipeline and configuration provided in the repository

- No additional data augmentation or architectural modifications

### Evaluation Results

When evaluating the model trained on ScanNet on the KITTI-360 test set, I obtained the following results:

```
2025-12-28 16:11:57,805 - INFO - Averaged metrics, num_queries: 1.0000,
avg_voxel_iou_1: 0.1941, avg_point_iou_1: 0.1933, mask_bce_1: 0.3533, mask_dice_1: 0.7933,
avg_voxel_iou_2: 0.4215, avg_point_iou_2: 0.4191, mask_bce_2: 0.2348, mask_dice_2: 0.5455,
avg_voxel_iou_3: 0.5110, avg_point_iou_3: 0.5072, mask_bce_3: 0.1901, mask_dice_3: 0.4421,
avg_voxel_iou_4: 0.5601, avg_point_iou_4: 0.5560, mask_bce_4: 0.1687, mask_dice_4: 0.3895,
avg_voxel_iou_5: 0.5877, avg_point_iou_5: 0.5850, mask_bce_5: 0.1509, mask_dice_5: 0.3608,
avg_voxel_iou_6: 0.6106, avg_point_iou_6: 0.6079, mask_bce_6: 0.1483, mask_dice_6: 0.3411,
avg_voxel_iou_7: 0.6298, avg_point_iou_7: 0.6267, mask_bce_7: 0.1414, mask_dice_7: 0.3232,
avg_voxel_iou_8: 0.6434, avg_point_iou_8: 0.6404, mask_bce_8: 0.1364, mask_dice_8: 0.3128,
avg_voxel_iou_9: 0.6434, avg_point_iou_9: 0.6394, mask_bce_9: 0.1379, mask_dice_9: 0.3088,
avg_voxel_iou_10: 0.6499, avg_point_iou_10: 0.6453, mask_bce_10: 0.1355, mask_dice_10: 0.3066
```

### Issue

Compared to the cross-dataset results reported in the paper, these numbers are significantly lower.

I would like to ask:

1. Is this performance degradation expected when directly transferring a model trained on ScanNet to KITTI-360 without domain adaptation?

1. Are there any additional preprocessing steps, dataset-specific configurations, or training tricks (e.g., class remapping, coordinate normalization, voxel resolution adjustments) that are required to reproduce the paper’s results?

1. Were the paper’s cross-dataset results obtained using fine-tuning, mixed-dataset training, or any extra supervision not explicitly mentioned?

### Additional Information

- I verified that the dataset paths and splits are correct.

- The evaluation code and metrics computation follow the official implementation.
Any clarification or guidance would be greatly appreciated.
Thank you very much for your time and for releasing this codebase.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.