DeepReefMap Paper Clarification

Hi @josauder and @HuguesSib,
Thanks for open sourcing this amazing project !
It is super useful both for the computer science and marine science community !

I have a question about the original paper
(Scalable semantic 3D mapping of coral reefs with deep learning).

I haven't yet explored the whole code implementation, and I have a question about the semantic integration in the methodology. In section 2.3.3 you describe how the semantic is integrated in the 3D model, but the explanation is not very detailed with respect to the implementation details.

What are the (low level) steps to build the explicit semantic map?

• Are you masking input images and then using the semantically segmented dataset to run a parallel learning based SfM with the masked RGB images instead of the original colored images?
• How are the semantic labels integrated in the semantic point cloud? Do you have an additional fourth dimension with the class label for each 3D point?
• How do you handle semantic inconsistency from different viewing directions? The segmentation model is not perfect, in different frames the same feature point can be part of different classes. What if the same 3D point is classified as soft coral in one frame and rock in another? Are you integrating any consistency constrain?
• Is the semantic knowledge somehow helping the SfM learning or not? Is it just a masking the 3D model after 3D reconstructing?

Sorry if the question are a bit vague, but I'm working in a similar project for deep sea data (using 3DGS) and I'm highly interested about the details on how you are integrating semantic in an underwater 3D reconstruction pipeline. Also the most evident and obvious detail can be very helpful for me ;)

If you prefer to talk about it more in details outside of the issue section, I would really appreciate your help.
Don't hesitate to contact me and thanks again!

The issue names no repository files, tests, or entry points. Start by reading section 2.3.3 of the linked paper and tracing its semantic integration claims through the repository implementation. Done means documenting the low-level pipeline, label integration, consistency handling, and whether semantics affect SfM, with references to the relevant code.