0
$\begingroup$

Rosanswers logo

I have the following dependencies in my package.xml

<buildtool_depend>catkin</buildtool_depend>

<build_depend>message_filters</build_depend>
<build_depend>rosbag</build_depend>
<build_depend>rospy</build_depend>
<build_depend>tf</build_depend>

<build_export_depend>message_filters</build_export_depend>
<build_export_depend>rosbag</build_export_depend>
<build_export_depend>rospy</build_export_depend>
<build_export_depend>tf</build_export_depend>

<exec_depend>message_filters</exec_depend>
<exec_depend>rosbag</exec_depend>
<exec_depend>rospy</exec_depend>
<exec_depend>tf</exec_depend>

<exec_depend>python-h5py</exec_depend>
<exec_depend>python-opencv</exec_depend>
<exec_depend>python-pandas</exec_depend>
<exec_depend>python-tqdm</exec_depend>

I have done rosdep install my_package and catkin build inside my workspace (and I got no error). However, whenever I try to do pip list or to import any of the Python dependencies I specified above, such as Pandas or OpenCV, I get an ImportError or I don't see them in the list. I want to use those Python dependencies in my scripts or source files inside my_package.

If the correct way of specifying Python dependencies is by specifying them in <exec_depend> (see e.g. http://docs.ros.org/lunar/api/catkin/html/howto/format2/python_module_dependencies.html)), then why can't I see pandas when doing pip list?


Originally posted by nbro on ROS Answers with karma: 372 on 2019-09-22

Post score: 0


Original comments

Comment by gvdhoorn on 2019-09-22:
What is the output of which python and python --version?


Edit: I doubt <build_export_depend>rospy</build_export_depend> is needed: rospy is a pure Python package. It cannot be a build_export_depend as there are no headers for other (C++) packages to depend on/include.

In general: I would guess 99% of Python dependencies would be only exec_depends.

Comment by nbro on 2019-09-22:
The output of the first command is /home/me/catkin_ws/src/my_package/venv/bin/python and of the second is Python 2.7.15+. I am inside a virtual environment, as you can see.

Comment by nbro on 2019-09-22:
Regarding the <build_export_depend>rospy</build_export_depend>, it was added automatically when I used catkin create pkg --catkin-deps rospy rosbag tf message_filters -- my_package.

Comment by gvdhoorn on 2019-09-22:
Yes, it does that. But that is just because there is no support for Python packages specifically. You can remove all dependencies other than exec_depends from your manifest. Unless you have a mixed C++/Python package, that is.

Comment by nbro on 2019-09-22:
@gvdhoorn How do I know which packages contain only Python code? Now, you told me that rospy only needs to be specified as a exec_depend, because it only contains Python code. In general, though, is there a way of knowing whether a dependency only contains Python code, without having to look at the source code?

Comment by gvdhoorn on 2019-09-22:
"only contain Python code" is just a shortcut.

build_export_depend is only used for telling packages that depend on you that they also (transitively) depend on your exported dependencies.

So if you have a header (C++) that includes a.hpp, which comes from package pkg_a, then in order for dependants of your package to be able to include your headers, they must also have pkg_a, or your headers will fail compilation with a cannot find 'pkg_a/a.hpp'.

Python is not a compiled language, so this cannot happen. Ergo: no build_export_depends.

Unless you have a mixed C++/Python package, have C++/C bindings or are doing other more complex things.

Comment by nbro on 2019-09-22:
@gvdhoorn However, you did not answer my specific question above about "how do I determine whether a package only needs to be specified as exec_depend, only contains Python code, or is or not required at build time. Maybe I will ask another question on the website.

Comment by gvdhoorn on 2019-09-22:
You really need me to explain you how to determine that a package "only contains Python code"?

There is no single command or fool proof way to see that from anything other than by looking at all the sources that make up the package.

Also: you add build_export_depends to your package. As you are -- presumably -- the author of your own source code, you should know whether you are #include-ing anything from your dependencies in your own headers. You should also know whether you are creating a Python-only package.

build_export_depends communicate to your users which packages your package is using. Not the other way around.

Comment by nbro on 2019-09-22:
@gvdhoorn Well, what would happen if a new version rospy contained C++ code? Then this would break some code. There should be a way of tagging packages as Python-only, etc., packages, so that this can be looked up easily, and people would know how to specify the dependencies. ROS is one of the worst pieces of software that I've ever seen, given its quite unnecessary complexity and bad design. I hope that ROS2 will improve all these points.

Comment by gvdhoorn on 2019-09-22:\

ROS is one of the worst pieces of software that I've ever seen, given its quite unnecessary complexity and bad design.

I would not make such blanket statements without being aware of the history that made some of these design choices necessary.

I hope that ROS2 will improve all these points.

Why "hope"? If you have concrete suggestions to improve things, contribute. That way you can make sure it will be better than what you now consider "unnecessarily complex and bad[ly] design[ed]".

It's a community project. Without community contributions it will not go into the direction a community member/the community will want. It will go into the direction the contributors want it to. Which may not be what you want.

Comment by gvdhoorn on 2019-09-22:\

Well, what would happen if a new version rospy contained C++ code? Then this would break some code. There should be a way of tagging packages as Python-only, etc., packages, so that this can be looked up easily, and people would know how to specify the dependencies.

Please read my comment again: build_export_depend says something about your package.

If your package is a Python-only package (read: your code, not your dependencies), then you don't need to list any build_export_depends.

If your package contains source code that is compiled, uses an inclusion system and requires dependants to have access to resources you include (again: in your source code), and those resources need to be around at build time, only then do you list them as build_export_depends.

I have the feeling you believe you need to check all your dependencies and then state whether they should be build_export_depends. You don't.

If you cannot #include it, it cannot be an export depend.

Comment by gvdhoorn on 2019-09-22:
Additionally:

Well, what would happen if a new version rospy contained C++ code? Then this would break some code

if/when that happens, that would be a breaking change.

Those would not be introduced without ample communication and a tick-tock approach.

If such things change, yes, it may break packages. But that is why breaking changes are typically handled with care.

This would be no different from breaking changes in any other software product or project, so is not ROS specific.

Comment by nbro on 2019-09-22:
@gvdhoorn Ok, thanks! Now, I understand which dependencies I have to specify in build_export_depend. However, to specify a dependency as a build one, I still need to look at the source code of those dependencies, but this is a drag.

$\endgroup$

1 Answer 1

0
$\begingroup$

Rosanswers logo

The output of the first command is /home/me/catkin_ws/src/my_package/venv/bin/python and of the second is Python 2.7.15+. I am inside a virtual environment, as you can see.

This is most likely the cause of you not being able to import the packages listed in the package manifest: rosdep will use your OS' package manager -- which in the case of Ubuntu (and Debian) would be apt -- to install the system dependencies that you've listed in your package's manifest.

For python-pandas for instance, it would lookup the rosdep rule for that key (here) and resolve it to python-pandas (in this case the rosdep key resolves to the same Ubuntu/Debian package name, but that is a coincidence).

It would then ask apt to install the python-pandas Debian package. On Ubuntu (and Debian), that would result in the Python 2 version of that package getting installed in the default system location: /usr/lib/python2.7/dist-packages/pandas. There is also a Python 3 version of the pandas rosdep key (here), but that wouldn't change what happens (it would just install the Python 3 version of Pandas in the system default location).

apt does not take any virtual environments into account, it will just install in the system default location (or really: it installs files in whichever location is specified in the .deb package).

This immediately means that the packages are installed, so rosdep will be happy (as apt reports it has installed whatever rosdep asked of it), but the packages will not be available in your virtual environment.

However, whenever I try to do pip list [..] I don't see them in the list.

this should be clear now: pip will look in your virtual environment, and as the packages have not been installed there, it will not list them as available.


Edit:

is there a way of installing the dependencies in the virtual environment with a ROS tool or I need to e.g. install the dependencies manually (from e.g. a requirements.txt file)?

rosdep will only use apt on Ubuntu/Debian and pip for certain dependencies (typically recognisable by the -pip suffix on their rosdep key). As Python virtualen environments don't wrap/redirect apt, nor where apt installs Python modules, I would classify this as a shortcoming of how Python virtual environments work (combined with how rosdep prefers apt on Ubuntu/Debian and how Ubuntu/Debian prefers to package Python modules as .deb packages).

There is no way I know of that would let you list just the Python dependencies of a ROS package. You could perhaps use rospack rosdep $pkg_name to get a list of system dependencies (which Python pkgs typically are), somehow filter those to leave just the Python dependencies, and feed that list to the pip in your virtual environment.

I've never done something like that, so I have no idea how well that would work.

Providing a requirements.txt to your users (which would duplicate some of the dependency information from the package manifest) would certainly be convenient in cases where they'd like to use a virtual environment.


Originally posted by gvdhoorn with karma: 86574 on 2019-09-22

This answer was ACCEPTED on the original site

Post score: 2


Original comments

Comment by gvdhoorn on 2019-09-22:
The same 'problem' can arise with any Python interpreter other than the system default one btw. It's not unique to virtual environments.

Comment by nbro on 2019-09-22:
Yes, you are apparently right. If I deactivate the virtual environment, I can now see the Python dependencies and import them. However, what if I did not want to pollute my global Python distribution while developing, is there a way of installing the dependencies in the virtual environment with a ROS tool or I need to e.g. install the dependencies manually (from e.g. a requirements.txt file)?

Comment by gvdhoorn on 2019-09-22:
just remembered there is actually a way to have rosdep output rules only for specific installers. You can pass it the --filter-for-installers option with as value pip for instance. This only works with the db command though.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.