missing pairs in pair selection

I noticed an issue when selecting pairs during feature matching, find_best_altitude returns very large value and hence causing missing pairs.
Here attached the dumped data from my dataset (I'm not allowed to shared the source data). One is origin, the other is directions, which is the exact input for find_best_altitude
https://drive.google.com/file/d/1dYMLx7NsPB8-hYLdarJkrwikSzOej2il/view?usp=sharing

All the images are nadir images, they are pointing down, some of the directions are exact (0,0,-1), while some of them are not but very closed to (0,0,-1), probably due to precision at compute time. The samples_x and samples_y plots like below, as you can see, in order to fit a 2nd degree equation and find the minimum/maximum point, you can get a very large or small extrema, negative extrema will be ignored, so we only consider very large value here. In the above example, I get 8136309.134196892. When multiply this to the directions that are not exact (0,0,-1), for example, (-1.56556042e-05, -6.45686431e-05, -1.00000000e+00), which will get (-1.27378835e+02, -5.25350441e+02, -8.13630913e+06). You can see there will introduce several hundreds of horizontal offsets, this will make it not able to be paired with images that it suppose to be. I got 2 sub reconstructions that are not aligned with each other.

https://github.com/mapillary/OpenSfM/blob/c798da16206aa52395eaef09ea4d7746028479d3/opensfm/pairs_selection.py#L74-L102

Start in opensfm/pairs_selection.py at lines 74-102 and inspect find_best_altitude. Reproduce the large extrema with the linked dumped origin and directions data, including directions close to (0,0,-1). Done means pair selection no longer loses expected pairs because of the large value and the affected sub-reconstructions are aligned.