How do robotics companies track faults in production systems?

Question

Are there any specific tools or practices which robotics companies use in order to keep track of errors in:

1. Sensors/Actuators malfunctioning or returning unexpected results
2. Programming errors or exception states

The aim of tracking this diagnostics information would be to:

1. Remote diagnosis of errors
2. Root cause analysis
3. Understanding of errors which 'cascade' from devices/robots to platforms and cloud services

I'm interested specifically in the parallels between normal software tools, e.g. Sentry or Rollbar for exception tracking, APM systems such as Dynatrace and the distributed and long lived state of the autonomous systems world. These systems seem to be better suited to errors in a request/response format, where the history of a device does not have as much affect on it's future as in the autonomous systems world.

So far the only tool I've seen is the diagnosis tools built into the ROS messaging bus, however it does not seem well suited to production.

How are teams currently doing this in production?

Answer 1

Company size and market expectations shape the specific tools used. Almost always kept confidential to some degree.

Issue tracking, remote diagnosis (and device security), and stochastic monitoring of hardware and software are common practice.

Failure prediction modeling is common where the economics make failure costly.

There are no specific tools and best practice is sector specific.

You might gain some insight looking at the practices of academic projects that require a large number of robots to be created and shared with other universities (iCub for instance); or by looking at robotics companies that have a strong open source commitment. But projects like these are biased by what tools the project leaders are willing to pay for.

If you are part of a team in a new company with no experience in this you are probably better off hiring an expert or two in your sector than reinventing 100 wheels.

EDIT Note that robotics hardware products can take 5 to 10 years to develop into a mature product, be deployed for another decade or two, and best practices will be from the era of development.