How do I render h264 video data in Foxglove Studio?. My data is recorded as a separate file alongside an MCAP file. Ideally I'd like to be able to open my MCAP file and then attach my video and have them time synced so they're both controlled by the scrubber.
$\begingroup$
$\endgroup$
1
-
$\begingroup$ Have you looked into storing h264 in mcap directly, which could solve the scrubbing integration (mostly)? I'm trying that out with rosbags and putting per-frame h264 data into individual ros messages, and although I need to run a decoder to view it it's working decently- ideally foxglove could decode it natively in your case. When you scrub it would take a second or two to get a usable image out of a decoder, depending on key frame frequency. $\endgroup$– Lucas WalterCommented Dec 15, 2022 at 14:43
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
To answer my own comment above integrated h264 in foxglove is an open issue https://github.com/foxglove/mcap/issues/427 (and more discussion https://github.com/foxglove/community/issues/87) - so it looks to be on their road map but not yet supported.
I'm not familiar with foxglove but you could possibly write an external application (or your own plugin?) that receives a clock/timestamp output from foxglove when you move the scrubber or play (do they support that?) and then indexes into your video, which you could have pre-processed for the location of frames (especially index frames)- and you'd need to have timestamps that can match up with what foxglove is using embedded in your video or in another file alongside it. It would publish out decoded images in a format foxglove can show live.
PyAV (https://github.com/PyAV-Org/PyAV and python3-av in Ubuntu) could be an easy way to get a basic external version working (but it doesn't support nvidia gpu decoding- though I'm not clear on that).
answered Dec 15, 2022 at 15:10
$\begingroup$
$\endgroup$
Foxglove now has built-in support for viewing H.264 video recorded in MCAP files (or any other data source Foxglove supports) with the CompressedVideo schema, as announced in this blog post:
…
These NAL units must be written to your log in a format that Foxglove will understand. For this, we’ve created the
foxglove.CompressedVideomessage schema – set the format field to “h264”, and thedatafield to contain the NAL units in Annex B format.Once these CompressedVideo messages have been recorded to an MCAP file, drag and drop the file into Foxglove, add an Image panel to your layout, and open the panel settings to select your video topic (e.g. /camera)
It's important to note that the data must be in Annex B format!
You can also try it out using the iOS WebSocket bridge app which can publish H.264-compressed video to Foxglove from an iPhone camera (source code here).
