# IMU-Camera Senor Fusion

I am working on fusing IMU and Camera Sensor Fusion for the Drone to precisely land on the target location.

With the Camera, I am tracking the April Tag which is on the ground. This gives me the x,y,z position of the camera in the drone with respect to the April tag on the ground.

With the IMU, I am measuring the Acceleration and Angular rates of the drone. From which I am integrating and finding the x,y,z position of the drone with respect to the starting point.

My question is,

1. In my case, the Camera provides the x,y,z position with respect to April Tag. And IMU provides x,y,z position with respect to its starting point. How these two different position information fused together help in improving the position accuracy of my system ??

2. Can I use the Camera x,y,z position to reduce the drift in the IMU. Or Should I only use GPS information to reduce the drift in IMU ??

In my case, the Camera provides the x,y,z position with respect to April Tag. And IMU provides x,y,z position with respect to its starting point. How these two different position information fused together help in improving the position accuracy of my system ??

With an Extended Kalman Filter(EKF). IMU for short term prediction step, and Camera measurements for the slower April Tags position updates. The IMU x,y,z positions are supposed to be integrated from the latest position, not some arbitrary starting point.

Can I use the Camera x,y,z position to reduce the drift in the IMU. Or Should I only use GPS information to reduce the drift in IMU ??

Yes. And you should use both.

Longer discussion:

The way this is properly done is with an EKF. The IMU gives you position information at a much faster rate in the prediction step portion. The camera, and gps measurements update the EKF estimate at a much slower rate. You can find a ton of papers,tutorials and code online just looking for IMU+GPS EKF.

Should also show you that the way you are describing using the IMU is wrong. You integrate for x,y,z information from the IMU from the last known good position(e.g your latest camera measurement), not from some arbitrary starting point. After a couple of seconds integrating the IMU is already quite inaccurate, so can't use it from some far away position anyways.

The place where you do have a coordinate system mismatch is with the camera and the GPS. GPS gives you GPS coordinates and camera information is relative to the April tag. For this you have 2 options. For both you have to choose a main coordinate frame. For these examples I am going to pick the camera($$c$$).

We had a gps measurement at $$p_1^g$$ which coincided with our latest position $$p_1^c$$. We now get a new gps measurment $$p_2^g$$. We can now calculate the relative transform from the first gps to the second. $$p_2^g-p_1^g=p_{2,1}$$. And then update the position in the camera frame using this relative transform. $$p_2^c=p_1^c+p_{2,1}$$.
Maybe you know where the April tag is in GPS coordinates. Or the first time you observe the April tag you record the GPS and camera position. You can then cal offset between camera and GPS frames. So that when you get a GPS measurement $$p_2^g$$ you can calculate the position in the camera frame $$p_2^c=p_2^g+offset$$.