Example for Estimating the Fundamental Matrix

The [Fundamental Matrix](https://en.wikipedia.org/wiki/Fundamental_matrix_(computer_vision)), $\boldsymbol{F} \in \mathbb{R}^{3 \times 3}$, is the matrix which given a set of correspondence between 2 images, $\left\\{ \left( \boldsymbol{p}\_{i}, \boldsymbol{q}\_{i} \right) \right\\}_{i = 1}^{n}$ , obeys:

$$ \boldsymbol{p}\_{i}^{\top} \boldsymbol{F} \boldsymbol{q}\_{i} = 0, \\; \forall i $$

The Fundamental Matrix must have Rank 2 and by definition is defined up to a scale.

The above can be solved by:

$$ \arg \min_{\boldsymbol{F}} \sum\_{i = 1}^{n} {\left( \boldsymbol{p}\_{i}^{\top} \boldsymbol{F} \boldsymbol{q}\_{i} \right)}^{2}, \\; \text{ subject to } \\; \boldsymbol{F} \in \mathcal{R}_{2}^{3}, \\; {\left\\| \boldsymbol{F} \right\\|}\_{F} = 1 $$

Where $\mathcal{R}\_{2}^{3}$ is the set of a $3 \times 3$ matrices with rank 2: $\mathcal{R}\_{2}^{3} = \left\\{ \boldsymbol{A} \in \mathbb{R}^{3 \times 3} \mid \text{rank} \left( \boldsymbol{A} \right) = 2 \right\\}$.

I wonder if such manifold exist?

This is opened after discussion with @kellertuer .

**Remark**
An extension with solving it with the objective $\arg \min_{\boldsymbol{F}} \sum\_{i = 1}^{n} \left| \boldsymbol{p}\_{i}^{\top} \boldsymbol{F} \boldsymbol{q}\_{i} \right|$ can make it more robust and even a paper worth.