What software libraries are there for assisting the general problem of parsing a stream of sensor data?
We use various sensors like LIDARs and GPSINS units that provide messages in proprietary binary formats, and have to write drivers for each one. Even though there's a lot of similar concepts used in each sensor (like a general purpose datagram for all messages, consisting e.g. of start/end sentinels, length specifications and a checksum, and then a variety of well-defined message formats for the payload), it ends up being a lot of tedious work to develop a driver each time.
I'd love a solution where I can write out packet/message specifications in some format, and have a library that finds & extracts valid messages from a stream, and provides them in a simple data structure format.
I'm not too fussed about what language, but basically want a general purpose datagram parsing library. There's a lot of customisation with sensors, maybe some odd format parsing, and probably some initial configuration to start the data stream, so this is really something I want as a library for processing the data in real-time that can be used as part of a driver/application.
Everything I find is either too basic (the low level tools for interpreting individual elements, so still lots of time spent extracting individual elements explicitly), or too specific (i.e. parsers written specifically for one particular protocol).
As a concrete example, consider NMEA messages:
- There's a basic outer datagram (starts with
$followed by message name, then comma separated data, and ends with
*, checksum and line terminating character)
- Data is in ASCII so needs to be parsed to binary for computational use
- Outer datagram allows for validation and removal of incomplete/corrupted messages
- Message name & content would be further parsed for consumption
- Field names can be specified for ease of use
A 'GPGLL' message might be turned from
$GPGLL,4533.21,N,17739.11,W,113215.22,A*31 into a programmatic data structure containing latitude, longitude, UTC timestamp and its validity.