0
$\begingroup$

Rosanswers logo

###Question: What would be the expected outcome of deleting logs while a process is running?

Specifically which of the following categories would that fall under : ) ?

  • Fine
  • Undefined
  • Very Bad

###Use Case: Our logs are growing too big because of a bug in diamondback where roscpp_internal does not respect log levels. As a work around we have a script that deletes particular log files ever 30 seconds or so. We've also seen a crash (assert/-6) that we've yet to diagnose closely following our log workaround change, and we currently suspect our log deletions as the cause.


Originally posted by Asomerville on ROS Answers with karma: 2743 on 2012-07-02

Post score: 2


Original comments

Comment by Eric Perko on 2012-07-02:
Maybe it's a silly question but, could you upgrade to Electric or Fuerte to fix that bug? :)

Comment by Asomerville on 2012-07-02:
Unfortunately the system where this is occurring is a production machine which is locked to diamondback for the time being.

Comment by joq on 2012-07-02:
Not silly, but I think we do still need to be able to build, test and release Diamondback fixes.

Comment by Thomas on 2012-07-02:
If you need help with debugging this assertion, could you please try to to send more information about what happens? (a backtrace would be a good start)

Comment by Asomerville on 2012-07-02:
Thanks for the offer. Because of the nature of the situation (remote machine not internet connected) we're going treat-first-diagnose-later. I'll update if we find anything interesting.

$\endgroup$

2 Answers 2

0
$\begingroup$

Rosanswers logo

Actually what you try to do just will not help. If you get a look at /proc/XXX/fd where XXX is the process id of your ROS node, you will be able to see the currently opened files and in particular the log files you thought were deleted.

Don't forget that file descriptors are a mechanism of reference counting on a resource, so as long as rosconsole maintains the file opened, the file will, in fact, not be deleted.

This explains why what you do is perfectly safe on Linux :)

So maybe this assertion is just due to you consuming the whole disk space and making the process crash or something? Monitor your disk space, you will see it decreasing until your process crashes, the file descriptor gets released and the kernel cleans the space keps by the previous log files...

So I think it would be better to try to really fix the error instead of trying to workaround it if possible ;)


Originally posted by Thomas with karma: 4478 on 2012-07-02

This answer was ACCEPTED on the original site

Post score: 2


Original comments

Comment by joq on 2012-07-02:
@Thomas is right, that is how POSIX filesystems work. The space will not be recovered until all references to the inode are gone.

Comment by Asomerville on 2012-07-02:
Duly noted regarding file deletions while the fd is still held.

$\endgroup$
0
$\begingroup$

Rosanswers logo

I would expect it to work OK in terms of system stability.

But, the status of future log messages from that run might be undefined (e.g. they might get lost).

What do you expect? What actually happens?


Originally posted by joq with karma: 25443 on 2012-07-02

This answer was NOT ACCEPTED on the original site

Post score: 1


Original comments

Comment by Asomerville on 2012-07-02:
Well we had a crash(assert/-6) in a system where we had been deleting the logs. We don't have enough info to do a postmortem, but the current assumption is that it was caused by our log deletion. I'll add some of this info to the original question.

Comment by joq on 2012-07-02:
Seems plausible. Maybe construct a small test case that logs continuously, while some other process periodically deletes the log?

Comment by Asomerville on 2012-07-02:
Hopefully we'll get a chance to do some more tests, but it may instead end up being a treat-first-diagnose-later sort of situation do to time/budget constraints.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.