Software libraries for parsing sensor data

Question

What software libraries are there for assisting the general problem of parsing a stream of sensor data?

We use various sensors like LIDARs and GPSINS units that provide messages in proprietary binary formats, and have to write drivers for each one. Even though there's a lot of similar concepts used in each sensor (like a general purpose datagram for all messages, consisting e.g. of start/end sentinels, length specifications and a checksum, and then a variety of well-defined message formats for the payload), it ends up being a lot of tedious work to develop a driver each time.

I'd love a solution where I can write out packet/message specifications in some format, and have a library that finds & extracts valid messages from a stream, and provides them in a simple data structure format.

I'm not too fussed about what language, but basically want a general purpose datagram parsing library. There's a lot of customisation with sensors, maybe some odd format parsing, and probably some initial configuration to start the data stream, so this is really something I want as a library for processing the data in real-time that can be used as part of a driver/application.

Everything I find is either too basic (the low level tools for interpreting individual elements, so still lots of time spent extracting individual elements explicitly), or too specific (i.e. parsers written specifically for one particular protocol).

As a concrete example, consider NMEA messages:

• There's a basic outer datagram (starts with $ followed by message name, then comma separated data, and ends with *, checksum and line terminating character)
• Data is in ASCII so needs to be parsed to binary for computational use
• Outer datagram allows for validation and removal of incomplete/corrupted messages
• Message name & content would be further parsed for consumption
• Field names can be specified for ease of use

A 'GPGLL' message might be turned from $GPGLL,4533.21,N,17739.11,W,113215.22,A*31 into a programmatic data structure containing latitude, longitude, UTC timestamp and its validity.

Answer 1

YACC and Lex to the rescue.

They are exactly what you ask for: "a general purpose datagram parsing library". Bytes go in one end, and structured data comes out the other end. You are responsible for doing 2 things:

Tokenizing

Lex is used for tokenizing – deciding how the input stream of bytes should be grouped. For example, to tokenize the text of a book, you'd want the tokens to be the words (i.e. all the letters and the apostrophe), and the punctuation. In the GPS message you showed, the tokens would be the literal string $GPGLL (and separately, any other GPS commands that start with $), commas, groups of digits, N, W, and a few others. This is essentially regular expression matching.

Parsing

YACC is used for parsing – deciding how the sequence of tokens should form structure. To use the example of text in a book, parsing would allow you to tell whether certain text was a quotation (or parenthetical text) by keeping track of the opening and closing marks. As an aside: try as you might, there is no regular expression that can tell you whether quotes or parenthesis are evenly matched; hence the need for parsing. In the GPS example, the parse rules would be something like "a GPGLL command is the $GPGLL token followed by ,, followed by a float (which itself is an integer token followed by a . optionally followed by another integer), followed by an N or S token"... and so on.

Writing a parser is essentially writing a context-free grammar.

Although these may sound scary and difficult (they certainly sounded so to me...), I can say from personal experience that learning how to use these tools is much much much easier than trying to write your own parser from scratch and then maintain it. There are many examples and tutorials online that should help you get started.

Answer 2

I have a feeling you won't get a generic solution for you problem. The reason for that is that you normally need to keep some sort of state on your device. Finding a general form to handle all different cases will end up in a construct that is likely quite complex. So why not use the general purpose tool that you have to handle such complexity (e.g. C++ or whatever other language you use).

That being said, there are some generic parts you could factor. In Rock for Example, we derive most drivers from a base library called iodrivers_base. This would do all the annoying IO stuff, and you just need to implement package parsing, package assembly and your state machine.