Practicality of Rolling Shutter Reconstruction

Hello,

I have been talking for a while with the folks over at OpenDroneMap about how to add rolling shutter camera reconstruction to their workflow. Their user community is very interested in this use case, as most drone-based camera systems are rolling shutter. After digging into their codebase a bit, I found that they are leveraging OpenSFM for the photogrammetric workflow, so in addition to the work needed on their end, there would need to be support for a rolling shutter camera model in OpenSFM for the work to be practical. So, I wanted to open an issue here to see what the scope of the effort would be in order to add such support.

In my experience, adding this type of camera model is a multi-part effort. Normally it would require the following:

1. The actual camera model. For a drone application where there is a stabilizing gimbal, rolling shutter distortion primarily comes from aircraft motion. As such, I think a linear model would likely be sufficient.
2. Since the camera model has more DoF, more feature points are required and those feature points need to span the FOV of the camera in order to stably fit the motion parameters. In order to get more point, I have found it either requires more aggressive parameters on an existing feature extractor or a new extractor.
3. If epipolar constraints are used in confining feature matching, then those tolerances either need widened or a new constraint to account for an initial estimate of the rolling shutter distortion needs developed. If using global matching, nothing needs to change.
4. Since from 2 there are a lot more matches in the system, tie points need filtered down to eliminate mismatches and ensure a good spread of tie points in the space of each image. I am not sure if OpenSFM support blocking, but if so block sizes tend to have to be smaller for rolling shutter problems.
5. The bundle adjustment needs to be adjusted to support the new camera model.

Have you guys scoped this effort out at all previously? If not, could we start a discussion on this front? I would be interested in helping in its development.

Thanks,
Phil

Start by mapping the existing camera-model and reconstruction workflow, then assess the five areas named in the issue: feature extraction, matching constraints, tie-point filtering, and bundle adjustment. The work is done only when a linear rolling-shutter model is supported throughout the OpenSfM workflow and its effects on matching and reconstruction are validated.