How do I render h264 video data in Foxglove Studio?. My data is recorded as a separate file alongside an MCAP file. Ideally I'd like to be able to open my MCAP file and then attach my video and have them time synced so they're both controlled by the scrubber.
$\begingroup$ Have you looked into storing h264 in mcap directly, which could solve the scrubbing integration (mostly)? I'm trying that out with rosbags and putting per-frame h264 data into individual ros messages, and although I need to run a decoder to view it it's working decently- ideally foxglove could decode it natively in your case. When you scrub it would take a second or two to get a usable image out of a decoder, depending on key frame frequency. $\endgroup$– Lucas WalterDec 15, 2022 at 14:43
To answer my own comment above integrated h264 in foxglove is an open issue https://github.com/foxglove/mcap/issues/427 (and more discussion https://github.com/foxglove/community/issues/87) - so it looks to be on their road map but not yet supported.
I'm not familiar with foxglove but you could possibly write an external application (or your own plugin?) that receives a clock/timestamp output from foxglove when you move the scrubber or play (do they support that?) and then indexes into your video, which you could have pre-processed for the location of frames (especially index frames)- and you'd need to have timestamps that can match up with what foxglove is using embedded in your video or in another file alongside it. It would publish out decoded images in a format foxglove can show live.
PyAV (https://github.com/PyAV-Org/PyAV and
python3-av in Ubuntu) could be an easy way to get a basic external version working (but it doesn't support nvidia gpu decoding- though I'm not clear on that).