facebookresearch / facebookresearch/easy3d
Performance gap when training on ScanNet and testing on KITTI-360
- Dominant language
- Python
- Stars
- 28
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
### Description
Hello, thank you for the excellent work on this project and the accompanying paper.
I am writing to report a significant performance gap that I observed when reproducing cross-dataset evaluation results, and I would like to ask for clarification on whether this behavior is expected.
### Experimental Setup
- Training dataset: /data/agile3d/ScanNet
- Testing dataset: /data/agile3d/KITTI360
- Training procedure:
- Followed the official training pipeline and configuration provided in the repository
- No additional data augmentation or architectural modifications
### Evaluation Results
When evaluating the model trained on ScanNet on the KITTI-360 test set, I obtained the following results:
```
2025-12-28 16:11:57,805 - INFO - Averaged metrics, num_queries: 1.0000,
avg_voxel_iou_1: 0.1941, avg_point_iou_1: 0.1933, mask_bce_1: 0.3533, mask_dice_1: 0.7933,
avg_voxel_iou_2: 0.4215, avg_point_iou_2: 0.4191, mask_bce_2: 0.2348, mask_dice_2: 0.5455,
avg_voxel_iou_3: 0.5110, avg_point_iou_3: 0.5072, mask_bce_3: 0.1901, mask_dice_3: 0.4421,
avg_voxel_iou_4: 0.5601, avg_point_iou_4: 0.5560, mask_bce_4: 0.1687, mask_dice_4: 0.3895,
avg_voxel_iou_5: 0.5877, avg_point_iou_5: 0.5850, mask_bce_5: 0.1509, mask_dice_5: 0.3608,
avg_voxel_iou_6: 0.6106, avg_point_iou_6: 0.6079, mask_bce_6: 0.1483, mask_dice_6: 0.3411,
avg_voxel_iou_7: 0.6298, avg_point_iou_7: 0.6267, mask_bce_7: 0.1414, mask_dice_7: 0.3232,
avg_voxel_iou_8: 0.6434, avg_point_iou_8: 0.6404, mask_bce_8: 0.1364, mask_dice_8: 0.3128,
avg_voxel_iou_9: 0.6434, avg_point_iou_9: 0.6394, mask_bce_9: 0.1379, mask_dice_9: 0.3088,
avg_voxel_iou_10: 0.6499, avg_point_iou_10: 0.6453, mask_bce_10: 0.1355, mask_dice_10: 0.3066
```
### Issue
Compared to the cross-dataset results reported in the paper, these numbers are significantly lower.
I would like to ask:
1. Is this performance degradation expected when directly transferring a model trained on ScanNet to KITTI-360 without domain adaptation?
1. Are there any additional preprocessing steps, dataset-specific configurations, or training tricks (e.g., class remapping, coordinate normalization, voxel resolution adjustments) that are required to reproduce the paper’s results?
1. Were the paper’s cross-dataset results obtained using fine-tuning, mixed-dataset training, or any extra supervision not explicitly mentioned?
### Additional Information
- I verified that the dataset paths and splits are correct.
- The evaluation code and metrics computation follow the official implementation.
Any clarification or guidance would be greatly appreciated.
Thank you very much for your time and for releasing this codebase.
Contributor guide
Assessment
This issue has not been assessed yet.