0
$\begingroup$

Rosanswers logo

I have read through http://wiki.ros.org/Get%20Involved as well as the links on it, but they don't give a black and white answer as to whether:

  1. Automated tests are required (ie, unit tests, integration tests, etc)
  2. Code reviews will be done prior submission of a release package

For (1), I was thinking it is required, however, I do find existing official packages without unit test frameworks.

Reason for this post is, due to timeline constraints, I wanted to know the "minimal" requirements needed for "officially releasing a ROS package", where "minimal" is less than the requirements of an "ideal" or "highly recommended."

Note: ROS package concerned is ROS1 Melodic, for Ubuntu 18.04.

Please advise.


Originally posted by mmrm on ROS Answers with karma: 3 on 2020-06-01

Post score: 0


Original comments

Comment by gvdhoorn on 2020-06-01:
Could you clarify what you mean by "official ROS package"? ROS is a community project, but you seem to imply that there are packages which are special in that project, in that they are considered official.

Comment by mmrm on 2020-06-01:
Hi @gvdhoorn, by "official ROS package", I meant those which are already Indexed as part of ROS packages itself, and has been released via bloom. To be more specific, I meant "official ROS package" as those available at https://www.ros.org/browse/list.php.

Comment by gvdhoorn on 2020-06-01:
So over 90% of that list is community contributed. The fact they are released through Bloom does not make them official. It just means the author(s)/maintainer(s) ha(s)(ve) made the effort to make them available as binary packages in a certain ROS distribution.

Some people assume the packages maintained/released by OR are official, and while they are certainly core packages and almost always used, I don't consider them any more official than others.

Comment by gvdhoorn on 2020-06-01:
As to https://www.ros.org/browse/list.php: you may want to use the ROS Index instead. It contains almost the same list of packages, but much more information about each of them.

$\endgroup$

1 Answer 1

0
$\begingroup$

Rosanswers logo

I meant those which are already Indexed as part of ROS packages itself, and has been released via bloom. To be more specific, I meant "official ROS package" as those available at https://www.ros.org/browse/list.php.

being released with Bloom and having binary packages built does not make a package an official one.

It just means the author/maintainer have gone through the process of following the wizard and providing the convenience of being able to apt install their package(s) (or equivalent command on other OS).

For ROS 1, there aren't any requirements on specific tests, (static) analysis or quality metrics. There are best-practices and nice-to-haves, but no thresholds or quality gates. Releasing a package is at the sole discretion of the author, relying on his or her skill and experience as to which level of quality he or she would be able to achieve.

(this seems like a free-for-all, and it sort-of is, but it's very likely a package will not see many users, nor much usage, if it doesn't live up to the expectations about quality which potential users will have. In a way, it's a survival of the fittest approach)

There is something which could be considered an "official list" of packages though, which would be REP-150: ROS Melodic and Newer Metapackages (and the earlier versions of that REP). That REP lists the packages which are to be considered part of the default set of packages which come with the various ROS variants for a specific version of ROS (ros-core, ros-base and desktop-full fi).

However, there is still no hard quality gating for those packages (but they are expected to have reached a certain quality, if only by their age).

For ROS 2, this is different.

While there are still no gates or thresholds which bar releasing a package, there are guidelines and even requirements documented which outline when a package author/developer may claim a certain quality level. Those are described in REP 2004: Package Quality Categories:

This REP describes a set of categories meant to convey the quality or maturity of packages in the ROS ecosystem. Inclusion in a category, or quality level, is based on the policies to which a package adheres. The categories address version, change control, documentation, testing, dependency and platform support policies.

Again: releasing is up to the package author/maintainer, but claiming a package (or set of packages) to have reached a certain level of quality is testable against the requirements given in REP-2004.

(note: many of the requirements are subjective, so if you expected KPIs such as maximum cyclomatic complexity, you will not find those in the REP. Having tests, and which kind, and what the readme should contain, how vulnerability disclosure is handled is part of the REP, so it focuses a bit more on process than on hard, machine verifiable metrics)


Edit:

is there a rule that says all REP 2xxx are applicable only for ROS2 somehow?

No, I don't believe so. But it's at least how I've interpreted those. Looking at the content, that would also make sense.


Edit 2:

It's just that I thought there would a bit of a "hard rule" on the requirements as to making the package qualified to be under https://www.ros.org/browse/list.php, especially on the Quality/Tests aspect.

No, there is none.

Also: that list is automatically generated based on the content of ros/rosdistro (ie: it's a machine generated list which contains all packages (ever) released). There is no human in the loop there, nor is it a curated list of some sort.

Specific context is, I plan to release a package, and I'm not sure if I'm even allowed to submit

"allowed" is a really bad word here. No one will stop you. You can release whatever you want (within reason, ie: obvious duplicates of core functionality, or obvious attempts at bad faith or malintent will most likely not be merged into ros/rosdistro).

For better or worse, as I wrote earlier, there are no hard requirements to the quality of ROS 1 packages. So on what grounds would anyone stop you from releasing your package?

or it will be rejected as soon as they see that my package doesn't have test codes.

Who would be "they"? As I wrote earlier: ROS is a community project. There is no "release committee" or similar body (which you seem to be implying).

The people reviewing Pull Requests to ros/rosdistro essentially just make sure the database is maintained in proper working order (ie: no entries added which will break the buildfarm, that sort of thing) and act as a first-level filter for the kind of bad-actor behaviour I described earlier.

There is no popularity contest or other form of ballotting.

I can't draft a project schedule not knowing this (I don't have unit/integration tests at the moment). That black and white judgment is what I'd like to know, if possible.

I hope it's now clear from my earlier comments and replies to your comments there is no one going to stop you from submitting your repository to ros/rosdistro (which is essentially what releasing means, although Bloom is typically used to do it).

Have you submitted a package to https://www.ros.org/browse/list.php before?

No one has, as, as I wrote above, it's an auto-generated page based on the content of the ros/rosdistro repository.

Many ROS users have released a package, as is evidenced by the length of that page, but they do that using Bloom, which in the end, updates the database hosted at ros/rosdistro.

So summarising: as long as you're not attempting to integrate a trojan which will be used to mine bitcoins on the computers of all ROS users worldwide, you are free to release whatever you created.

Would it be good to reserve time in a project plan for tests? Absolutely.

Will your package(s) releasable without tests? Absolutely (as in: technically, yes. I can't say anything about the quality of the result in that case of course or the fitness for a particular purpose, but there will be no filters (human or automated) to block your submission.


Originally posted by gvdhoorn with karma: 86574 on 2020-06-01

This answer was ACCEPTED on the original site

Post score: 3


Original comments

Comment by gvdhoorn on 2020-06-01:
Note that there were questions about using/applying REP-2004 to ROS 1 packages as well.

While I don't believe there would be anything very ROS 2 specific about the requirements in REP-2004, it wasn't written with ROS 1 in mind.

Comment by mmrm on 2020-06-01:
Thank you @gvdhoorn. From your answer, I can extract that releasing via bloom is a separate topic from making the package available at https://www.ros.org/browse/list.php. With this, I would like to clear out that my intent at least for this question, is make my ROS1 package available at https://www.ros.org/browse/list.php.

And from what I can understand from your answer, unlike ROS2 with REPs even, there are no Quality or Test-related requirements related to making my ROS1 package available at https://www.ros.org/browse/list.php. Such requirements are discretionary to me as the developer. This means, as a developer, I can make do with just build tests and manual tests using actual hardware and gazebo simulation and still able to submit (and pass) for inclusion at https://www.ros.org/browse/list.php, even without unit test and integration test code/framework (ie, using rostest, gtest, unittest) as part of my submitted package.

Is my understanding of your answer correct?

Comment by mmrm on 2020-06-01:
link text

Oh. But is there a rule that says all REP 2xxx are applicable only for ROS2 somehow?

Comment by gvdhoorn on 2020-06-01:
It would be good if you could provide a little more context.

I could just write "yes" or "no", but without more context, such answers have a tendency to be misinterpreted.

Are you asking in general, or with a certain purpose or context in mind?

Comment by mmrm on 2020-06-01:
Hmm. It's just that I thought there would a bit of a "hard rule" on the requirements as to making the package qualified to be under https://www.ros.org/browse/list.php, especially on the Quality/Tests aspect. Specific context is, I plan to release a package, and I'm not sure if I'm even allowed to submit or it will be rejected as soon as they see that my package doesn't have test codes. I can't draft a project schedule not knowing this (I don't have unit/integration tests at the moment). That black and white judgment is what I'd like to know, if possible. Have you submitted a package to https://www.ros.org/browse/list.php before?

Comment by mmrm on 2020-06-01:
Additionally, at http://wiki.ros.org/Quality/Tutorials/UnitTesting, I saw a line saying "This is why, when you are contributing code to ROS, whether bug fixes, or new features, you have to include automatic unit tests. Unit tests are a required step in our QA process."

However, I saw lots of packages under the https://www.ros.org/browse/list.php without test codes. So now I'm confused what is really the rule for ROS1 regarding automated tests.

Comment by mmrm on 2020-06-01:
Thanks so much for all the details @gvdhoorn. Everything is clear to me now, I get the whole picture and the info that I need. :)

Comment by gvdhoorn on 2020-06-02:
No problem.

$\endgroup$

This site is temporarily in read-only mode and not accepting new answers.