1
$\begingroup$

Rosanswers logo

I'm running melodic on a ubuntu 18.04 container (through systemd-nspawn inside Archlinux:

uname -a: Linux rsl-bionic 5.3.8-arch1-1 #1 SMP PREEMPT @1572357769 x86_64 x86_64 x86_64 GNU/Linux

Using only rosrun rosout rosout Uses upwards of 8.6Gb of Resident memory. I don't have the ponts to upload the traces, also I'm not really sure what I'm doing when I am trying to trace it output. But in one allocation 8.6Gb seems to be initialized in the XmlRpcServer in a

std::vector<pollfd, std::allocator<pollfd> > ::resize(unsigned long) in ?? (libroscpp.so)

Any pointers for fixing/tracing this? I'll probably have to give up and repartition my machine and install ubuntu natively, but I can't reason as to why/how it could be related.


Originally posted by ixil on ROS Answers with karma: 23 on 2019-11-06

Post score: 2

$\endgroup$

1 Answer 1

1
$\begingroup$

Rosanswers logo

I am running the same setup and ran into the same problem but managed to fix it after creating a Debug build of the core ros libraries and analyzing the code of the XmlRpcServer where the massive memory allocation happens.

The XmlRpcServer in ros_comm sets up a vector for querying all possible file descriptors to check if it has run into the limit of opened files. When running a container on Arch however, this limit is 1073741816. With each file descriptor having a size of 8 Byte, thats 8GB for every ros node. You can check the current (hard) limit of the running shell with ulimit -Hn. If you reduce the limit with ulimit -Hn 524288, the memory usage should be fine again.

You can make this limit persistent by adding the file /etc/security/limits.d/30-nofilelimit.conf in the container with the following content:

#<domain>  <type>  <item>  <value>

*          soft    nofile  1024
*          hard    nofile  524288

Originally posted by Ralino with karma: 46 on 2019-12-19

This answer was ACCEPTED on the original site

Post score: 3


Original comments

Comment by gvdhoorn on 2019-12-20:
This is some good detective work @Ralino.

I would strongly suggest you open an issue on ros/ros_comm about this. Please provide sufficient detail about how you found out and where the vector is created, and a link to this Q&A.

Seeing as the focus of development is on ROS 2 these days I cannot guarantee it will be fixed (or implemented differently), but it would be tremendously valuable to have a record of your findings on the issue tracker regardless.

Comment by ixil on 2019-12-20:
@Ralino thanks for taking the time to post your work. I gave up in the end using the container approach. But relieved to know there is a solution

Comment by pumaking on 2023-03-22:
If anyone is having trouble with setting the hard limit: check out https://askubuntu.com/questions/1061572/unable-to-modify-ulimit, you may also need to run ulimit -Sn 524288.

Comment by danambrosio on 2023-03-27:
You can also use docker run ... --ulimit nofile=1024:524288 to set the soft and hard nofile limits. If you want this for all containers consider adding:

"default-ulimits": {
    "nofile": {
        "Name": "nofile",
        "Hard": 524288,
        "Soft": 1024
    }
}

To your /etc/docker/daemon.json file.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.