Calibration of Spherical Images together with Perspective Images

Hi,

I'm trying to calibrate 360 images captured from Go Pro Max with regular perspective images and the feature matching stage fails some percentage of the time (mostly in indoors settings). This, I believe, is because feature extraction does not differentiate Spherical images from Perspective ones and so the feature descriptors are quite different due to the warped textures near the poles of spherical images compared to the perspective ones. This ends up generating so few matches between the 360 and perspective images.

I'd like to implement a new functionality to remedy this. My idea is while computing the features (& descriptors) of spherical images, I will compute cubemap images from sphericals and extract the features on the cubemap textures. I'll then map the pixel locations from cubemap to spherical locations so rest of the pipeline will stay the same. So, my feature descriptors will then be living in perspective space. I'm also thinking about rotating the spherical image ( 45 around x and 45 degrees around y axes ) and extract features on the new cubemap and augment the original set with these to also extract features around cubemap image boundaries.

This should also improve matching near the poles of the 360 images as well - I noticed that most matches close to the poles are also outliers between the 360-360 matches -- when the camera moves around, pole location changes and so different parts of the scene get warped. This cubemap based feature descriptor computation should improve things for that scenario as well.

It looks straightforward to me but I wanted to ask your opinion before I start implementing this - do you have any recommendations ( or is there something I need to be careful about here ? )

cheers!

Start by tracing the feature extraction and feature matching stages described in the issue. Define how cubemap-derived features would map back to spherical locations and how rotated cubemaps would augment them; done means improved matching for spherical-to-perspective and spherical-to-spherical cases without changing the rest of the pipeline.