I'm trying to get VREP vision sensor output processed with opencv via ROS api. I did manage to set up scene and get scripts running, but the problem is that I get somewhat like 4-5 fps even without actual processing (currently I just push images directly to output).
This issue does not seem to depend on image resolution, since both 1024512 and 128128 captures result in exactly the same fps.
This also is not a matter of blocking calls, although I'm posting single-theaded code, I do have quite complex multithreaded processing pipeline which performs rather well with actual cameras (~50 fps).
Lua scripts on VREP's side do not seem to be a problem also, since I've tried to play with video retranslation examples provided by vrep, and they seem to achieve rather satisfying fps.
So it seems like image streaming is a bottleneck.
Here's my sample script:
# coding=utf-8
import rclpy
import rclpy.node as node
import cv2
import numpy as np
import sensor_msgs.msg as msg
import third_party.ros.ros as ros
class TestDisplayNode(node.Node):
def __init__(self):
super().__init__('IProc_TestDisplayNode')
self.__window_name = "img"
self.sub = self.create_subscription(msg.Image, 'Vision_sensor', self.msg_callback)
def msg_callback(self, m : msg.Image):
np_img = np.reshape(m.data, (m.height, m.width, 3)).astype(np.uint8)
self.display(np_img)
def display(self, img : np.ndarray):
cv2.imshow(self.__window_name, cv2.cvtColor(img, cv2.COLOR_RGB2BGR))
cv2.waitKey(1)
def main():
ros_core = ros.RclpyWrapper()
node = TestDisplayNode()
rclpy.spin(node)
node.destroy_node()
rclpy.shutdown()
if __name__ == "__main__":
main()
I also have to mention that I run it with ros bridge, since I need processing done with python3, which is supported by ROS2 only, and VREP seems to work only with ROS1 (although I'm just starting to work with these systems, so I'm not confident in that case).
UPDATE:
It seems like I have bee mistaken a bit about ros bandwidth capabilities. Frame rate actually changes with frame resolution.
But what actually puzzles me is how it is actually supposed to work? The best result I could get is 6464 video with decent frame rate. Even two 6464 sensors result in lower frame rate. How machine vision dependent projects are supposed to be done in with ROS?
I was actually hoping to use simulation with ROS since it could provide a lot of background and simplify testing and evaluation. Although I have not try to use gazebo yet.
Any advice is welcome.
Copying some comments, since I need to remove my update, which was originally created as an answer post.
@William : I haven't tried sending images with rclpy -> rclpy , but our image demos in C++ perform a lot better than that on most machines. Maybe as a base line you could try one of our image demos to see if the issue exists there too: https://github.com/ros2/ros2/wiki/Quality-Of-Service
@William : Good performance with images (even large ones) is definitely possible with ROS 1 (lots of historical examples of this working well), and should be possible with ROS 2, but it is more likely to have a bug or performance problem that can be fixed upstream.
UPDATE (13.07.2018):
I have reviewed QoS a bit, but did not really get if it might be used with ros1 bridge or not.
Another thing which I tried was to remove ros bridge at all. So I wrote one more receiver with old interface and,although frame rate is rather low, it's way better already (approximately 15-20 fps). At least it looks smooth, which is quite enough for now.
Thus, VREP/ROS1 bridge works good enough, so does ROS1/python2 bridge (note, that I still use np.reshape manually, and output is in the same thread, so it might work faster).
I also assume that ROS2/python3 bridge works quite good. Actually what happens wth ros2 code, is that it gets each 5th or each 10th frame approximately (this actually might change a bit during single test run). I tried to remove all output and image transformations, leaving only header logging, which changed nothing.
Regrettably, I could not check it with showimage example, since it was failing after displaying somewhat similar to:
[INFO] [showimage]: Subscribing to topic 'image'
[INFO] [showimage]: Received image #78
I assume that ros1/2 bridge is responsible for frame loss. If anyone has any ideas on fixing or, at least, testing it, you're welcome.
UPDATE (15.07.2018):
I tried to change QoS settings. Setting KEEP_ALL policy fixes problem frameskipping problem. But it seems like images are being stuck in receiver buffer. Memory consumption just continues to grow. So, backend seems to receive everything, and messages are not being discarded by bridge. The problem is on rclpy side.
Originally posted by RomanShametko on ROS Answers with karma: 56 on 2018-07-09
Post score: 1
Original comments
Comment by William on 2018-07-09:
There are a lot of moving pieces here, with VREP/ROS 1 and the ros1_bridge and rclpy. It's unlikely anyone will have time to reproduce your entire pipeline and then profile it for you to find the issue. I'd recommend trying to narrow down which part is slow and make a simpler example from that.
Comment by William on 2018-07-16:
What version of ROS 2 are you using? I'd recommend using the Bouncy release or building ROS 2 from source using the latest "master".
Comment by RomanShametko on 2018-07-17:
I'm using latest distro (the one I found here https://github.com/ros2/ros2/wiki/Linux-Install-Debians). Actually, there's a bug created on github (https://github.com/ros2/ros2/issues/509). Currently I have MWE posted there, and I also plan to get some profiling done by today night.
Comment by marguedas on 2018-07-30:
Following up here as well:
One main difference between the C++ and Python messages is that in Python we enforce the validity of the data set in the message fields at runtime.
To skip the checking of data validity you can set PYTHONOPTIMIZE=0 in your environment. Performance should be much better