0
$\begingroup$

Rosanswers logo

Hi !

I am using Ubuntu 16.04 with a real-time patched kernel (4.14.93-rt53). I have tested my system with cyclic tests and am happy about the realtime performance.

I have built and installed Ros2 (Crystal Clemmys) according to this manual: https://index.ros.org/doc/ros2/Installation/Linux-Development-Setup/. The talker and listener examples work fine.

When I try to run the pendulum demo (https://index.ros.org/doc/ros2/Tutorials/Real-Time-Programming/#run-the-tests) I get an unexpected output:

$ RMW_IMPLEMENTATION=rmw_connext_cpp pendulum_demo
RTI Data Distribution Service Non-commercial license is for academic, research, evaluation and personal use only. USE FOR COMMERCIAL PURPOSES IS PROHIBITED. See RTI_LICENSE.TXT for terms. Download free tools at rti.com/ncl. License issued to Non-Commercial User [email protected] For non-production use only.
Expires on 00-jan-00 See www.rti.com for more information.
RTI Data Distribution Service Non-commercial license is for academic, research, evaluation and personal use only. USE FOR COMMERCIAL PURPOSES IS PROHIBITED. See RTI_LICENSE.TXT for terms. Download free tools at rti.com/ncl. License issued to Non-Commercial User [email protected] For non-production use only.
Expires on 00-jan-00 See www.rti.com for more information.
Couldn't set scheduling priority and policy: Operation not permitted
Killed

I could get rid of the line:

 Couldn't set scheduling priority and policy: Operation not permitted

I have tried to give the rights according to this: https://index.ros.org/doc/ros2/Tutorials/Real-Time-Programming/#setting-permissions-for-the-scheduler

I can also see (after logon/logoff) that the rights are updated accordingly:

$ ulimit -r
98

Now the output is without this line, but still gets killed:

$ RMW_IMPLEMENTATION=rmw_connext_cpp pendulum_demo
RTI Data Distribution Service Non-commercial license is for academic, research, evaluation and personal use only. USE FOR COMMERCIAL PURPOSES IS PROHIBITED. See RTI_LICENSE.TXT for terms. Download free tools at rti.com/ncl. License issued to Non-Commercial User [email protected] For non-production use only.
Expires on 00-jan-00 See www.rti.com for more information.
RTI Data Distribution Service Non-commercial license is for academic, research, evaluation and personal use only. USE FOR COMMERCIAL PURPOSES IS PROHIBITED. See RTI_LICENSE.TXT for terms. Download free tools at rti.com/ncl. License issued to Non-Commercial User [email protected] For non-production use only.
Expires on 00-jan-00 See www.rti.com for more information.
Killed

I have found a very similar (unanswered) question and initially posted mine (by mistake) as an answer here: https://answers.ros.org/question/303072/ros2pendulum-demo-freezes-the-system/?answer=314834#post-id-314834

I would be happy to receive any Input. Since I am inexperienced with Ros, it might very well be some rookie mistake. Thank you! Nicolas

Update: Testing with rttest and debugging with gdb

According gvdhoorn's suggestion I have tested example_loop from ros2/realtime_support/rttest. Unfortunately this program gets Killed as well.

To debug the example_loop with gdb I have rebuild and started with

$ cmake ../ -DCMAKE_BUILD_TYPE=Debug 
$ make

$ gdb ./example_loop
(gdb) r

and received the following output:

Starting program: /home/<myUserName>/realtime_support/rttest/examples/build/example_loop
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program terminated with signal SIGKILL, Killed.
The program no longer exists.
(gdb) bt
No stack.
(gdb)

to check my user's permission (Thanks a lot for your support, I appreciate any guesse/suggestions!):

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 30272
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 98
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 30272
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

So far I have only tested it on this machine. I have created the Kernel according to this instruction Building-Realtime-rt_preempt-kernel-for-ROS-2.

I have used the latest stable RT_PREEMT patch (patch-4.14.93-rt53.patch.gz from http://cdn.kernel.org/pub/linux/kernel/projects/rt/4.14/, linked by https://wiki.linuxfoundation.org/realtime/start). I have applied this patch to the Kernel source linux-4.14.93.tar.xz, which I have taken from https://mirrors.edge.kernel.org/pub/linux/kernel/v4.x/ .


Originally posted by Nicolas Gerig on ROS Answers with karma: 11 on 2019-02-06

Post score: 1


Original comments

Comment by gvdhoorn on 2019-02-06:
Can you test whether example_loop from ros2/realtime_support/rttest does work? If that also gets killed, then something is wrong.

You might want to see whether gdb can help clear up the mystery in that case.

Comment by gvdhoorn on 2019-02-06:
Btw: your question title still mentions the privilege issue, but in your latest output I don't see it any more.

Comment by Nicolas Gerig on 2019-02-07:
Hi gvdhoorn, Thank you for your support! I have updated the title and tested the example_loop. Unfortunately, the example loop gets killed as well:

./example_loop
Killed

Sorry for taking so long to test your suggestion.

Comment by gvdhoorn on 2019-02-07:
First thing I'd try is run it in gdb.

Comment by Nicolas Gerig on 2019-02-07:
I am trying to do that now (never used gdb before). I have rebuilt example_loop to add debug info: $ cmake ../ -DCMAKE_BUILD_TYPE=Debug $ make Started the debugging by $ gdb ./example_loop It prints infor about gbd, then reads the symbols, and stays with (gdb). I dont see any CPU usage.

Comment by gvdhoorn on 2019-02-07:
Once gdb has started, you type r (short for run) and press enter.

It will probably start outputting all sorts of info. After it gets killed, copy-paste all that into your question (use the edit button/link).

If possible, also type bt and press enter, after it's killed.

Comment by gvdhoorn on 2019-02-07:
re: your edit: it could be that your user doesn't have the permission to request the large stacks that realtime_support is trying to use. What is the output of ulimit -a?

Note: I'm not sure here, just guessing at this point, as your program just getting killed isn't very informative.

Comment by gvdhoorn on 2019-02-07:
Also: have you tried running the same program on a different machine? How did you create the kernel?

Comment by tfoote on 2019-02-07:
A common readon for "Killed" messages is that you ran out of memory. You should check dmesg to see if the OOM killer was activated. It creates a log there.

Comment by Nicolas Gerig on 2019-02-07:
Thank you for your hints! I will investigate memory usage (and install the setup on a different machine as well). Unfortunately, I will be out of the lab for the next 10 days - so this thread will be put on hold and I will update as soon as I can continue testing.

Comment by gvdhoorn on 2019-03-04:
@Nicolas Gerig: please post your last edit as an answer and then accept your own answer. That will show much more clearly that your question was resolved/answered.

8GB of RAM, which is not enough according to the example manuals.

could you also link & quote from the documentation you refer to?

$\endgroup$

1 Answer 1

0
$\begingroup$

Rosanswers logo

Found the issue, my bad.

Thank you all for your valuable feedback! I have learnt a lot already while testing the below steps. However, my failure was rather trivial - the PC I have borrowed from our IT did only have 8GB of RAM, which is not enough according to the example tutorial of the pendulum_demo it requires at least 8GB of free RAM (from the Run the tests section):

Before you run make sure you have at least 8Gb of RAM free. With the memory locking, swap will not work anymore.

I failed to check the PC specs and the requirement in the example manual. However, it is interesting to note that not only the pendulum_demo, but also example_loop from ros2/realtime_support/rttest required more RAM than was available in the tested PC. By using htop in a seperate console I could follow how the memory is filled after either programm start and the programm being killed exactly once the memory is full.


Originally posted by Nicolas Gerig with karma: 11 on 2019-03-04

This answer was ACCEPTED on the original site

Post score: 0

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.