0
$\begingroup$

Rosanswers logo

I have the following scenario. We have a legacy codebase, all in one git-repository, with different modules and a legacy build system (in fact there are 4 different build systems already, 2 of them in daily used, qmake on Linux and vs in Windows). I want to create catkin packages for the individual modules and release them into a private rosdistro.

Because the code base will continue to be used with the legacy build system in non-ROS context, I have limited freedom to restructure and change things. I would like to be as unintrusive to the current layout and structure as possible, in order to not be faced with acceptance problems in parts of our team that are not using ROS (yet).

I have gotten as far as creating the catkin packages, making sure they work with the install space (see also a previous question) and releasing them with bloom into a private rosdistro. Now I want to build debians for the released packages (first manually, later with buildbot-ros or eventually with a private buildfarm setup). I am confronted with some issue that I will describe in the following.

The repository layout is as follows:

foo-project
├── config.h
├── external-libraries
│   └── libbar
│       └── bar.h
├── module1
│   ├── package.xml
│   ├── CMakeLists.txt
│   └── ...
├── module2
│   ├── package.xml
│   ├── CMakeLists.txt
│   ├── module2.h
│   └── ...
├── module3
│   ├── package.xml
│   ├── CMakeLists.txt
│   └── ...
├── ...
...

1. Header files outside the package directory

The first issue is that there is a header file config.h in the project root. It is included with #include "../config" in the headers of module1 (which are in turn included by other modules). I install the config.h together with the headers of module1, but I still need it to be in-reach at build-time of module1, which is not the case for the bloom-released branches, that contain only the module folder, not its parent. How could I inject this file into the bloom-released branch?

2. External library files at build time

In module1, the third party library libbar is included. It needs to be only there at build time, and not installed / exported. In the current module build-definition there is a include_directories(../external-libraries/libbar) and then in the source files of module1 it is referenced simply as #include "bar.h". Similar to the previous issue, I need to make these header files available at build time. But the difference is, that I don't need install them, and that they are not in the repository root, but in a parallel subfolder. This would make it possible to wrap them up in a foo_external_libs-dev catkin package.

3. Interdependencies between modules

Mostly the dependencies between the different modules can be mapped to a tree structure, with module1 at the root. This is reflected in the dependencies as declared in the package.xml files. However, to build the individual modules, there are unfortunately cyclic interdependencies. E.g. to build module1, module2.h is needed. When building in a catkin workspace, this works because the CMakeLists.txt of module1 has a include_directories(../module2). It is also fine with the install space, since it is only valid use that part of module1 in a third module, if you also depend on module2 explicitly. Similar to the above issues, when building the debian release, the relative path ../module2 is not valid. Again, I would want to somehow inject the needed header files from module2 into the release branch of module1 when releasing with bloom.

For issue 1, I could maybe achieve moving the config.h inside the module1 folder, for issue 2 I could possibly introduce a new catkin package foo_external_libs-dev, but for issue 3 I am at a loss, on how to properly achieve this without duplicating files in the source repository.

Any hints or ideas are much appreciated!

Edit: What would help me would be a way to tell bloom to execute a specific script in the full source repository before splitting it up into the separate packages. That way I could copy around the files to make sure each of the package directories contains all the files that are needed at build time.

Edit2: In response to @William's answer: I have already resorted to a preprocessing script, your statement "that bloom can import from any vcs branch or tag and even take archive files directly" is key here. My workflow now looks as follows:

I have a separate catkin-release branch. It contains a script for preprocessing the repository to address issues 1, 2 and 3. For all of them that entails copying some files into the respective modules, and changing some #include and include_directories statements as well as tweaking install rules. A little cp and sed is enough for my needs. For a new release I perform these steps:

  1. merge the latest release branch into catkin-release
  2. call the script to update the copied files if they have changed; commit
  3. do the standard steps for releases with bloom, where the upstream branch to be imported is now catkin-release --> catkin_prepare_release, then bloom-release

As you say, one could imagine incorporating the custom "prepare-release-script" into the bloom import-upstream step quite easily. However, since I am not very familiar with bloom, I'm not 100% sure how this would look like. In particular, what remains a mystery for me so far is which of the branches in the release repository are meant to be updated manually, and which are meant to be updated only by bloom? I would imagine that the branches with patches are meant to be changed manually to include the needed patches (is there documentation describing this? If so, I didn't find it yet.), so that seems to be a natural place where one might put custom scripts that would be called on top of applying the given patches. Does that make sense?

As for your other suggestions addressing the individual issues, I'm not sure how I could do what you describe in the upstream repository with the restrictions I mentioned (i.e. that I need to avoid changing the layout and the other build systems as much as possible).

What you suggest for issue 1 could probably be done, but only by also changing the other build systems. Might be acceptable though, since it is a very minor change.

I'm not sure I understand how your suggestion for issue 2 would work. How can I install the header external-libraries/libbar/bar.h from module1 if the file itself is not available in the imported release-repository-branch for module1? As mentioned in OP, I could place an additional catkin package in the external-libraries folder, which would then do this install step.

Thanks for your suggestion on issue 3, breaking up the dependency cycle. That is enlightening, but I fail to see how I can achieve that without changing the folder layout of my upstream repository, since I cannot fit two catkin packages in the module1 folder, or can I? With my current pre-processing script I simply copy the needed files from module2 into the module1 folder. I could of course also create this additional module1-headers package in the preprocessing script, but that seems overkill, since the current solution works well enough, even if it is not 100% clean (since certain headers of the module1 debian will only work if one also explicitly has installed the module2 debian, and there is no implicit dependency mechanism ensuring that).

For now I am happy with the workflow described above, so I will accept your answer. Feel free to comment on the questions above about parts of your answer, or not. I will leave extending bloom for another day. I need to first get a buildbot or buildfarm setup going.

Hopefully I can demonstrate to the team the benefits of catkin and in particular the tooling around it, such that in the future the acceptance to changing the code-layout is higher, which might allow me to address some of the issues without a custom "prepare-release" script.

You have certainly addressed all my questions and as always in a very helpful way! Thanks William!


Originally posted by demmeln on ROS Answers with karma: 4306 on 2016-04-28

Post score: 3

$\endgroup$

1 Answer 1

0
$\begingroup$

Rosanswers logo

For issue 1 (and possibly issue 2), I'd say your best option is to preprocess the repository and copy the required files into the packages which need them, i.e. copy the top level config.h into each of the packages and change how the config file is referenced. For example, rather than headers doing #include "../config.h" do #include "config.h" and add the top level directory to the include paths for each module with -I.

For issue 2, you can also install the file that would normally be referenced with ../other_module/header.h either to a folder off the normal include path, e.g. <prefix>/include/impl/other_module/header.h and pass -I<prefix>/include/impl to modules which need it, and also change the include statement to be #include "other_module/header.h".

For issue 3, I think you have to break at least one of the packages into two packages so that you can break the cycle. For example, you could have module1 install it's headers in a module1_headers package. Since that package would not need to build anything it doesn't need to build depend on module2, but only run depend (or technically build_export_depend on it. Then you can have module2 depend on module1_headers, and module1 depend on module1_headers and module2.

There is currently no way to run a script on import with bloom, but that would be easy to implement and I'd consider a pr that adds that functionality. Another option is that bloom can import from any vcs branch or tag and even take archive files directly. So you could write a script which preprocesses your original repository (moving and copying files, adding package.xml and CMakeLists.txt if needed, etc) and then creates an archive of the result. Then you could pass that to bloom without having to extend bloom at all.

I think I attempted to address all of your questions, so hopefully there is something helpful in there. Let me know if you have more questions.

EDIT:

This is in response to @demmeln's edited question.

As you say, one could imagine incorporating the custom "prepare-release-script" into the bloom import-upstream step quite easily. However, since I am not very familiar with bloom, I'm not 100% sure how this would look like. In particular, what remains a mystery for me so far is which of the branches in the release repository are meant to be updated manually, and which are meant to be updated only by bloom? I would imagine that the branches with patches are meant to be changed manually to include the needed patches (is there documentation describing this? If so, I didn't find it yet.), so that seems to be a natural place where one might put custom scripts that would be called on top of applying the given patches. Does that make sense?

So I think that adding a "prepare-release-script" into the release repository that bloom runs is ok, so long as that script doesn't bring in new dependencies. For example, it shouldn't use rospy or be written in OCaml or rely on something setup on the local computer like ssh keys. Otherwise not just anyone could run bloom at a later point. This somewhat applies to using linux specific tools or options on common tools, i.e. I run bloom on OS X and would be sad if I couldn't bloom something because the other guy wrote a script that only works on Linux. Not a big deal, but something to think about.

My best guess as to what this would look like is a setting in the tracks.yaml and the script would live in the master branch.

There isn't good documentation on the different branches in bloom. I can summarize quickly:

  • bloom gets the source code and puts it in a tarball (or downloads a tarball directly)
  • bloom extracts the tarball on the upstream branch
  • bloom overlays any "patch step files" which live on the master branch and generally include package.xml for things that don't have one because the next step needs a package.xml to be there
  • bloom extracts the version from the package.xml(s) (the version in all package.xml's have to match)
  • bloom creates the upstream/x.y.z version tag based on the package.xml(s) version
  • bloom creates a branch for each package as release/<rosdistro>/<package name> if it doesn't exist
  • bloom removes any patches that were previously applied to the release branch (maybe none) and store then in the patches/release/<rosdistro>/<package name> branch as .patch files
  • bloom copies the contents of the folder for each package into the corresponding release branch from the upstream branch
  • bloom tries to reapply any patches from the release-patch branch on top of the new version's files
  • if they don't apply, bloom drops into a shell and lets you try to sort it out
  • bloom creates the release/<rosdistro>/<package name>/x.y.z-i version tag (this is what from source people get for the released version of a package); the i is the debian increment number, which increments each time you run bloom and the upstream version didn't change
  • the rest of the steps are debian or fedora specific, but follow a similar pattern of copying forward; I'll do debian for example:
  • bloom creates a debian branch called debian/<rosdistro>/<package name> if it doesn't exist yet
  • bloom removes any patches that were applied last time to the debian-patch branch, patches/debian/<rosdistro>/<package name>
  • bloom copies the patched content of the release branch to the debian branch
  • bloom places the debian folder (merging with one from upstream if it already exists) and leaves some files untemplated
  • bloom reapplies patches from the debian-patches branch
  • bloom creates a debian versioned branch called debian/<rosdistro>/<debian name>/<package name>, e.g. debian/kinetic/xenial/rviz, if it didn't already exist
  • bloom removes patches that were applied previously (if any) to the branch with the same name but prefixed with patches/
  • bloom copies the patched content from the debian branch
  • bloom reapplies the patches that were previously removed (if any), allowing you to try and fix them if they don't apply
  • bloom replaces the template files with the actual, expanded files, e.g. debian/rules.em -> debian/rules
  • bloom creates the final tag that the build farm uses, called debian/<debian package name>_x.y.z-i_<debian name>, e.g. debian/ros-kinetic-rviz_1.12.1-0_xenial

There are a few differences depending on how you run bloom, but that's the most probably set of events that a normal release goes through.

Now you can directly commit changes to a few of the branches:

  • release/<rosdistro>/<package name>
    • patches here apply to the source tarballs that users download when using rosinstall and propagate down the bloom stream to debian and fedora
  • debian/<rosdistro>/<package name>
    • patches here affect all debian versions, e.g. for kinetic it would be jessie, wily, and xenial; also many of the debian files are still template files
  • debian/<rosdistro>/<debian name>/<package name>
    • patches here affect only the debian/ubuntu distribution in the name

You probably shouldn't edit any of the other branches, including the patches/* branches. Those are managed by bloom.

I'm not sure I understand how your suggestion for issue 2 would work. How can I install the header external-libraries/libbar/bar.h from module1 if the file itself is not available in the imported release-repository-branch for module1? As mentioned in OP, I could place an additional catkin package in the external-libraries folder, which would then do this install step.

Thanks for your suggestion on issue 3, breaking up the dependency cycle. That is enlightening, but I fail to see how I can achieve that without changing the folder layout of my upstream repository, since I cannot fit two catkin packages in the module1 folder, or can I? With my current pre-processing script I simply copy the needed files from module2 into the module1 folder. I could of course also create this additional module1-headers package in the preprocessing script, but that seems overkill, since the current solution works well enough, even if it is not 100% clean (since certain headers of the module1 debian will only work if one also explicitly has installed the module2 debian, and there is no implicit dependency mechanism ensuring that).

For all of that, I was assuming that you have some ability to change the original layout, but in ways that don't interfere with how it worked before (only moving files and folder or changing include lines).

Sorry it took so long for me to do a follow up. Hopefully it was in time to be useful if not educational to future readers.


Originally posted by William with karma: 17335 on 2016-04-28

This answer was ACCEPTED on the original site

Post score: 3


Original comments

Comment by Arthur Queiroz on 2017-01-24:
I've been looking for sometime for a detailed overview of what exactly bloom does and how, so this answer was really enlightening. Shouldn't an explanation like this be part of the bloom wiki?

Comment by Arthur Queiroz on 2017-01-24:
The same goes for the basic usage example (i.e. generating deb packing files for a single ros package) described on https://github.com/ros-infrastructure/bloom/issues/254#issuecomment-37210878. This simple example would make bloom easier to understand upon the first contact with it.

Comment by demmeln on 2017-01-30:
Yeah both answers by William are really great. It would be great to add those to the wiki, or at least some links to the answers here (anyone can edit the wiki ofc)...

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.