# Kalman filter for vision based pose estimation: 'good' measurements not improving system covariance

I am starting off with a very simple Kalman filter for vision based pose estimation (PnP algorithm). The filter is inspired by the constant velocity model in this OpenCV tutorial, but I am ignoring roll, pitch and yaw for now and I am only estimating the XYZ pose.

As PnP is formulated as a non linear least squares problem, I have access to a covariance matrix from the solver I am using (Ceres), and I am using this matrix as an estimate of the measurement noise covariance matrix $R$ at each step. Process noise covariance $Q$ remains constant. My understanding of the filter is that if I obtain more and more 'good' measurements (with low $R$), the system covariance should recursively decrease; and even if the measurements worsen later, the posteriors should not worsen too much. So if I were to start from an area of bad measurements with covariance $P_1$, move to an area where I receive some good measurements and back to the area of bad measurements with covariance this time being $P_2$, $P_2 < P_1$.

On the other hand, in my formulation, my posterior covariance $P$ seems to be blindly following $R$: in my previous example of bad-good-bad areas, $P$ decreases and increases to almost the same extent, so I can't see the advantage of receiving good measurements reflected in the final covariance. On the other hand, if I reduce $Q$ even lower, I can see some variation in initial vs. final covariances, but the system is not trusting the measurements at all and is smoothing the poses out way too much. I am confused as to how to pick the best values of $Q$ and $R$, and mainly as to how I can write my filter in a way that it recognizes the advantage of getting good measurements.