What's the statistically and mathematically correct way to process lidar data so that I can find out a good current estimate of the closet obstacle and its rate change?
Should I be learning about Kalman filters or is that the wrong kind of processing?
I am supposing you have a mobile robot moving around and you want to track its position and/or the position of obstacles nearby. Then the answer is yes, you need to learn some filtering, more precisely you should learn about Extended Kalman Filter (EKF) since your measurement (the LIDAR measurement) is non-linear as it is defined by range and angle of the ray.
There are many resources online, but I leave this link to Slides from University of Freiburg about EKF.
Note: In the case you do not have time to learn filtering and you just want (for pure undefined experimental purposes) to do some closest obstacle avoidance in a simple way, you can still use the raw measurement of the LIDAR (assuming is precise enough for your application) in order to build a map with obstacles to avoid.
Hope this helps.
