Camera drifts downwards despite being still

I'm testing granite with a VIO dataset collected with realsense d435i. I've tested this dataset with basalt and I could track it without issues. But when I run the same dataset with granite (and using only the left camera) the camera seems to drift downwards from the beginning of the sequence despite it being still. The sequence starts with camera still and then a forward motion, which I presume is good enough for granite's monocular initialisation. If I start the sequence at the point the forward motion is happening the granite output pose follows the forward motion while drifting downwards as before. In both cases Granite eventually crashes with and error SO3::exp failed! omega: nan nan nan, real: nan, img: nan.

I'm doing these tests with a ros wrapper I made myself for both basalt and granite projects. This way I can feed the dataset from a rosbag and display the camera pose in rviz. The calibration and configuration files are essentially the same for basalt and granite, just adapting for a few different parameters in each project. I think I've setup the granite pipeline correctly, so I'll assume, for the moment, I could rule out my ros wrapper for granite as the cause of this downwards drift. In this case, would you have any advice for what could be the reason for this downwards drift? any configuration parameters that could be the cause? something related with gravity?

Start by reproducing the drift and SO3::exp failure with the RealSense D435i VIO dataset through the ROS wrapper, then compare the calibration and configuration files used for Granite and Basalt. Done means identifying the cause of the downward drift and preventing the NaN crash, with the result verified on the affected sequence.

Written by the indexing model from the issue text.