How do I render h264 video data in Foxglove Studio?. My data is recorded as a separate file alongside an MCAP file. Ideally I'd like to be able to open my MCAP file and then attach my video and have them time synced so they're both controlled by the scrubber.
To answer my own comment above integrated h264 in foxglove is an open issue https://github.com/foxglove/mcap/issues/427 (and more discussion https://github.com/foxglove/community/issues/87) - so it looks to be on their road map but not yet supported.
I'm not familiar with foxglove but you could possibly write an external application (or your own plugin?) that receives a clock/timestamp output from foxglove when you move the scrubber or play (do they support that?) and then indexes into your video, which you could have pre-processed for the location of frames (especially index frames)- and you'd need to have timestamps that can match up with what foxglove is using embedded in your video or in another file alongside it. It would publish out decoded images in a format foxglove can show live.
PyAV (https://github.com/PyAV-Org/PyAV and
python3-av in Ubuntu) could be an easy way to get a basic external version working (but it doesn't support nvidia gpu decoding- though I'm not clear on that).
These NAL units must be written to your log in a format that Foxglove will understand. For this, we’ve created the
foxglove.CompressedVideomessage schema – set the format field to “h264”, and the
datafield to contain the NAL units in Annex B format.
Once these CompressedVideo messages have been recorded to an MCAP file, drag and drop the file into Foxglove, add an Image panel to your layout, and open the panel settings to select your video topic (e.g. /camera)
It's important to note that the
data must be in Annex B format!