Can't send binary String messages in ROS Noetic via rospy

Question

I have to send binary String messages via the ROS message bus. That worked out perfectly with Python 2.7 and ROS Melodic, but fails with ROS Noetic. Here's some example code (of course I don't just want to send 'test' as binary data):

import rospy
from std_msgs.msg import String

rospy.init_node('test_pub')
pub = rospy.Publisher('bin', String, queue_size=10)

while not rospy.is_shutdown():
    msg = String()
    msg.data = b'test'
    pub.publish(msg)
    rospy.sleep(1.)

With Noetic that fails as soon as someone subscribes to the topic. The reason is that ROS tries to encode the data of String messages via .encode(utf-8) once it detects Python 3, which will of course fail for bytes.

Here's the error:

File "/opt/ros/noetic/lib/python3/dist-packages/rospy/msg.py", line 152, in serialize_message
    msg.serialize(b)
File "/opt/ros/noetic/lib/python3/dist-packages/std_msgs/msg/_String.py", line 56, in serialize
    _x = _x.encode('utf-8')
AttributeError: 'bytes' object has no attribute 'encode'

And the problematic passage in the String messages source:

def serialize(self, buff):
"""
serialize message into buffer
:param buff: buffer, ``StringIO``
"""
try:
  _x = self.data
  length = len(_x)
  if python3 or type(_x) == unicode:
    _x = _x.encode('utf-8')

Is there some way to work around that without modifying/destroying the binary data itself? Decoding the binary data first so that ROS can encode it again unfortunately does modify it.

And yes, I know that there exist the messages Byte / MultiByteArray, but therefore I would need to convert the binary data to/from an Integer array on all ends, which would add massive overhead and break compatibility with ROS Melodic clients and rosbags.

---

Originally posted by gm88 on ROS Answers with karma: 13 on 2021-03-22

Post score: 0

Answer 1

Unfortunately, I believe this is by design.

See wiki/msg, specifically the parts about how Python 2 and Python 3 treat string and uint8[] fields.

It's essentially as you already described: in Python 3, a string field will be mapped onto bytes. In Python 2, it'll be str.

Pedantic, but I believe this is the problem:

I have to send binary String messages via the ROS message bus

According to wiki/msg again, string fields are intended to contain ascii strings, not binary data. Those two are not the same, both semantically and structurally (yes, they both typically occupy a contiguous piece of memory, and both are byte/char based, but that doesn't make them identical).

The violation of the semantics of string you propose then causes the problems with the code on different versions of ROS: it just happened to work prior to Noetic because Python 2 was very lax in this regard. But based on the semantics and the field type description, the developers were justified in fixing it in Noetic (they were also forced by the Python 3 migration of course).

Note: this is not a snooty answer trying to put blame on you. I just wanted to clarify for future readers why caring about semantics of ROS message fields (and semantics of messages in general) is important. Even if "it fits", it does not mean it's proper use. Abusing message types can lead to situations where your assumption something will work is broken by a maintainer.

Is there some way to work around that without modifying/destroying the binary data itself? Decoding the binary data first so that ROS can encode it again unfortunately does modify it.

if you really want to send binary data in string fields, I believe the only proper way would be to first encode it as Base64, and then decode it again in all receivers.

It would still violate semantics, but it would let you use a string for something which is technically not an ascii string.

It would of course not help with this:

would add massive overhead and break compatibility with ROS Melodic clients and rosbags.

---

Originally posted by gvdhoorn with karma: 86574 on 2021-03-22

This answer was ACCEPTED on the original site

Post score: 1