I'm trying to use the VINS-Fusion package to obtain odometry for a quadcopter. The primary sensor is an Intel Realsense D435i. I've calibrated the camera and IMU extrinsics using Kalibr. IMU Calibration using imu_utils gives noise standard deviation values that are too low to work with so I had to use values that are ~10-100x the values obtained from calibration.
The system works fine with this configuration (apart from trajectory errors) in sunny outdoor conditions but lately we've been seeing instances of uncontrolled drift in the Odometry. It works fine during one flight and fails inexplicably in the next. Sudden changes in brightness might be the issue so I've been looking into improving the image quality but apart from this I'm out of ideas. I've come across issues on the OpenVINS repository where it's suggested that the IMU data of the D435i is faulty. How should I go about making the system more robust? Should I work on improving my calibration results or get a better sensor?