How to evaluate 3D Occupancy grid maps

Question

I would like to know how to go about evaluating 3D occupancy grid maps.

I have a SLAM system that produces a 3D OGM (in .bt format using octomap/octovis)

I also have a ground truth OGM in same .bt format.

How do I compare the accuracy of my map to the ground truth map in a qualitative and quantitative way?

Important notes:

• The two maps may not be the same scale.
• One map may be less dense than the other.

One method I have thought about using is MRPT's occupancy grid matching application This would require me to send both 3d maps as a message to the octomap_server node in ROS, get the resulting map in Rviz, save the image 2D image of each separately, and then somehow convert the images to MRPT's .simplemap file format, and then run MRPT's grid matching program on the two files.

Surely there is a better/more accurate way?

EDIT: So I did more research and another route I could go is Matthew's Correlation Coefficient (MCC). I could compare two maps and iterate over each cell to compare my result to a ground truth, counting the True and False Positives and Negatives. Only problem with this is that I have to assume that the two maps are the same scale, and also in the same orientation.

If you have any ideas on solving these scale and orientation issues don't be shy.

Answer 1

Actually, evaluating maps with different scales and densities happens to be part of my research for my PhD so here goes the two ways I can think of on the top of my head:

• Visually: it is always the first step but not the only one ;).

• Map matching methods as in this paper by Sören Schwertfeger. That sounds like the idea you had first and I think it would probably be the best way to go about it.

Sören Schwertfeger's method is based on the Voronoi diagram and thus goes away your scale and density problem by have some fuzzy limits on the angle and length of the edges of the Voronoi diagram. I used a similar method to match hand drawn sketch maps onto ground truth maps using Voronoi diagrams, and I did not consider at all the angle and length of Voronoi edges, just their order. That's probably the worst case of "maps with different scale and density you can have" and the matching was not half bad. Thus, I believe that should be a pretty solid solution, especially if you have occupancy grid maps; you'll be able to take in account angles in between Voronoi edges to get a better precision.

Answer 2

I'd be (pleasantly) surprised to hear that there is a way to get the output you want based on your available inputs, but this problem seems incomplete.

The most accurate map is the one that gets you to your precise destination. To quantitatively measure that accuracy, you'd have to use your map to navigate from point A to point B, then compare that final position to the one you'd get with the ground-truth map. It sounds like you only have point A (the origin) defined for both maps, so you have no way to measure point B's error.

At the very least, you would need to start with a set of points that are defined on both maps. (I say "points" meaning "2 or more"; I can't convince myself that a round-trip measurement -- starting and ending at the origin -- would properly explore the space.)