Are there any specific tools or practices which robotics companies use in order to keep track of errors in:
- Sensors/Actuators malfunctioning or returning unexpected results
- Programming errors or exception states
The aim of tracking this diagnostics information would be to:
- Remote diagnosis of errors
- Root cause analysis
- Understanding of errors which 'cascade' from devices/robots to platforms and cloud services
I'm interested specifically in the parallels between normal software tools, e.g. Sentry or Rollbar for exception tracking, APM systems such as Dynatrace and the distributed and long lived state of the autonomous systems world. These systems seem to be better suited to errors in a request/response format, where the history of a device does not have as much affect on it's future as in the autonomous systems world.
So far the only tool I've seen is the diagnosis tools built into the ROS messaging bus, however it does not seem well suited to production.
How are teams currently doing this in production?