# object level sensor fusion for multiobject tracking

I want to fuse objects coming from several sensors, with different (sometimes overlapping!) fields of view. Having object lists, how can I determine whether some objects observed by different sensors are in fact the same object? Only then I can truly write an algorithm to predict future state of such an object.

From literature I read those 4 steps:

1. Plot to track association (first update tracks estimates and then associate by "acceptance gate" or by statistical approach PDAF or JPDAF)
2. Track smoothing (lots of algorithms for generating new improved estimate, e.g.: EKF, UKF, PF)
3. Track initiation (create new tracks from unassociated plots)
4. Track maintenance (delete a track if was not associated for last M turns. also: predict those tracks that were associated, their new location based on previous heading and speed)

So basically I am questioning point 1, acceptance gate. For a single sensor I can imagine it can be just a comparison of xy position of object and sensor measurement, velocity with heading eventually. My case is however, I have already ready object lists from each sensor in every cycle, there are some algorithms how to merge informations about an object collected by different sensors (great source is e.g. here: http://www.mathworks.de/matlabcentral/fileexchange/37807-measurement-fusion-state-vector-fusion), but question is how to decide which objects should be fused, and which left as they were? Fields of view may overlap partly, not totally.

• There are many more ways to do point 1, and point 1 is what you need. You may have to get creative, but that's the task you have. – Josh Vander Hook Apr 10 '13 at 14:13

This is called "data association" in tracking literature. When you measure the position of an object, you need to know which object it was you measured. If you can estimate this probability, then you are free to choose the most likely association. This is a heavily researched topic, but boils down to Bayesian analysis.

Here's a simple way:

Assume we have two objects $o_1$ and $o_2$, an estimate of their position $x_1$ and $x_2$, and a measurement $z$. Suppose $z$ is just a position measurement. We'd like to update $o_1$ or $o_2$'s estimated position, but we have no idea with object we just measured. So, we find the most likely object, and update that.

1. Estimate $p(z|x_1)$ and $p(z|x_2)$. The Kalman filter gives you the tools to do this.

2. IF $p(z|x_1)>p(z|x_2)$ AND $p(z|x_1)>\gamma$, then update $x_1$ using $z$, the Kalman filter gives you the tools to do this as well.

3. ELSE IF $p(z|x_2)>p(z|x_1)$ AND $p(z|x_2)>\gamma$, then update $x_2$ using $z$, the Kalman filter gives you the tools to do this as well.

4. ELSE, no probability is greater than $\gamma$, so we drop the measurement. You can set $\gamma$ to zero if you don't care.

Two things: First, you also need to know how to initialize an object. That's why $\gamma$ is important. If the measurement doesn't seem to match any known estimates, you might have just detected a new object.

Second, I have a suspicion that your question will be heavily edited, so I'm not putting a lot of details in just now.

You have a different problem, but it is totally related. First, you want to fuse objects. This is as simple as estimating the probability that all measurements of object 1 and all measurements of object 2 are in fact of the same object. The algorithm is simple, but nearly intractable computationally. This is the same as track merger.

To really do this with objects in 3D, you need a model of the object which allows you to estimate the probability that two partially overlapping (or possibly just nearby) objects are in fact the same object). But this is exactly the same as the acceptance gate approach.

note

There are only a few really useful ways to mere measurements, and they are well defined. I strongly suggest if you wish to continue with abstract tracking and estimation tasks that you read Bar-Shalom's Tracking and Estimation book. Please get it from a library if you can. I just don't know of a better reference.

• you are right, I will shortly edit my question. I add knowledge I have, it can be a good source for the others later also. So please answer to my updated question, as if I asked you here. Regarding your 1.2.3.4., silly question maybe, but how does KF let me estimate probability of measurement conditioned on position? I thought measurement just comes and is compared with estimated in innovation part. – josh131 Apr 10 '13 at 9:05

This depends on a lot of things. What kind of sensors? Are they stationary? Are they at least fixed with respect to each other?

For example, if you have a system of multiple cameras that are fixed with respect to each other, you can calibrate them, so that you know their relative positions and orientations. Then not only will you be able to determine whether two detections in different cameras are the same object, but you will be able to determine its position in 3D.

Here is the classic paper on camera calibration. To actually do the calibration you can either use OpenCV or the Caltech Camera Calibration Toolbox for Matlab. OpenCV has an example application for camera calibration which is easier to use than the Caltech toolbox.

• Sensors are fixed with respect to each other. First I am interested in type radar. And then eventually how it would work in combination with camera, or laser. What you wrote about is stereo vision right? – josh131 Apr 9 '13 at 21:12
• Yes, this is stereo vision. There must be also ways to calibrate a system consisting of other types of sensors. Unfortunately, I know next to nothing about radar. – Dima Apr 9 '13 at 22:27
• Ok no problem. Further on I am about to move to cameras also, are there some references you recommend me to learn from, once I'm into it? – josh131 Apr 9 '13 at 22:32
• I have edited my answer. – Dima Apr 10 '13 at 15:54