0
$\begingroup$

Rosanswers logo

It seems different versions of python3-catkin-pkg-modules are distributed on packages.ros.org. Therefore, when I tried to install python3-catkin-pkg-modules, sometimes it fails, and sometimes succeeds.

Environment

  • Ubuntu 18.04
  • ROS Melodic

How to reproduce the problem

$ docker run --rm -it ubuntu:18.04 /bin/sh

# apt update && apt install -y lsb-release gnupg
# sh -c 'echo "deb http://packages.ros.org/ros/ubuntu $(lsb_release -sc) main" > /etc/apt/sources.list.d/ros-latest.list'
# apt-key adv --keyserver 'hkp://keyserver.ubuntu.com:80' --recv-key C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654
# apt update
# apt install -y python3-catkin-pkg-modules
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  docutils-common libfreetype6 libjbig0 libjpeg-turbo8 libjpeg8 liblcms2-2 libpaper-utils libpaper1
  libpng16-16 libtiff5 libwebp6 libwebpdemux2 libwebpmux3 multiarch-support python3-dateutil
  python3-docutils python3-olefile python3-pil python3-pkg-resources python3-pygments python3-pyparsing
  python3-roman python3-six sgml-base tzdata ucf xml-core
Suggested packages:
  liblcms2-utils docutils-doc fonts-linuxlibertine | ttf-linux-libertine texlive-lang-french
  texlive-latex-base texlive-latex-recommended python-pil-doc python3-pil-dbg python3-setuptools
  ttf-bitstream-vera python-pyparsing-doc sgml-base-doc debhelper
The following NEW packages will be installed:
  docutils-common libfreetype6 libjbig0 libjpeg-turbo8 libjpeg8 liblcms2-2 libpaper-utils libpaper1
  libpng16-16 libtiff5 libwebp6 libwebpdemux2 libwebpmux3 multiarch-support python3-catkin-pkg-modules
  python3-dateutil python3-docutils python3-olefile python3-pil python3-pkg-resources python3-pygments
  python3-pyparsing python3-roman python3-six sgml-base tzdata ucf xml-core
0 upgraded, 28 newly installed, 0 to remove and 16 not upgraded.
Need to get 42.0 kB/3179 kB of archives.
After this operation, 15.7 MB of additional disk space will be used.
Err:1 http://packages.ros.org/ros/ubuntu bionic/main amd64 python3-catkin-pkg-modules all 0.4.20-1
  404  Not Found [IP: 64.50.236.52 80]
E: Failed to fetch http://packages.ros.org/ros/ubuntu/pool/main/p/python3-catkin-pkg-modules/python3-catkin-pkg-modules_0.4.20-1_all.deb  404  Not Found [IP: 64.50.236.52 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

# apt install -y python3-catkin-pkg-modules # try again
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  docutils-common libfreetype6 libjbig0 libjpeg-turbo8 libjpeg8 liblcms2-2 libpaper-utils libpaper1
  libpng16-16 libtiff5 libwebp6 libwebpdemux2 libwebpmux3 multiarch-support python3-dateutil
  python3-docutils python3-olefile python3-pil python3-pkg-resources python3-pygments
  python3-pyparsing python3-roman python3-six sgml-base tzdata ucf xml-core
Suggested packages:
  liblcms2-utils docutils-doc fonts-linuxlibertine | ttf-linux-libertine texlive-lang-french
  texlive-latex-base texlive-latex-recommended python-pil-doc python3-pil-dbg python3-setuptools
  ttf-bitstream-vera python-pyparsing-doc sgml-base-doc debhelper
The following NEW packages will be installed:
  docutils-common libfreetype6 libjbig0 libjpeg-turbo8 libjpeg8 liblcms2-2 libpaper-utils libpaper1
  libpng16-16 libtiff5 libwebp6 libwebpdemux2 libwebpmux3 multiarch-support python3-catkin-pkg-modules
  python3-dateutil python3-docutils python3-olefile python3-pil python3-pkg-resources python3-pygments
  python3-pyparsing python3-roman python3-six sgml-base tzdata ucf xml-core
0 upgraded, 28 newly installed, 0 to remove and 16 not upgraded.
Need to get 41.8 kB/3179 kB of archives.
After this operation, 15.7 MB of additional disk space will be used.
Get:1 http://packages.ros.org/ros/ubuntu bionic/main amd64 python3-catkin-pkg-modules all 0.4.18-1 [41.8 kB]
Fetched 41.8 kB in 6s (7383 B/s)

Investigation

I can see that there are three addresses for packages.ros.org.

$ host packages.ros.org
packages.ros.org is an alias for ftp.osuosl.org.
ftp.osuosl.org has address 64.50.233.100
ftp.osuosl.org has address 140.211.166.134
ftp.osuosl.org has address 64.50.236.52
ftp.osuosl.org has IPv6 address 2605:bc80:3010::134
ftp.osuosl.org has IPv6 address 2600:3402:200:227::2
ftp.osuosl.org has IPv6 address 2600:3404:200:237::2

These two have python3-catkin-pkg-modules_0.4.20-1_all.deb.

However, 64.50.236.52 has python3-catkin-pkg-modules_0.4.18-1_all.deb, and when DNS selected this address, it fails.

You can confirm the problem by directly downloading python3-catkin-pkg-modules_0.4.20-1_all.deb.

$ docker run --rm -it ubuntu:18.04 /bin/sh

# apt update && apt install -y wget
# wget http://packages.ros.org/ros/ubuntu/pool/main/p/python3-catkin-pkg-modules/python3-catkin-pkg-modules_0.4.20-1_all.deb
--2020-05-18 11:38:32--  http://packages.ros.org/ros/ubuntu/pool/main/p/python3-catkin-pkg-modules/python3-catkin-pkg-modules_0.4.20-1_all.deb
Resolving packages.ros.org (packages.ros.org)... 64.50.236.52, 140.211.166.134, 64.50.233.100, ...
Connecting to packages.ros.org (packages.ros.org)|64.50.236.52|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2020-05-18 11:38:38 ERROR 404: Not Found.

# wget http://packages.ros.org/ros/ubuntu/pool/main/p/python3-catkin-pkg-modules/python3-catkin-pkg-modules_0.4.20-1_all.deb
--2020-05-18 11:38:40--  http://packages.ros.org/ros/ubuntu/pool/main/p/python3-catkin-pkg-modules/python3-catkin-pkg-modules_0.4.20-1_all.deb
Resolving packages.ros.org (packages.ros.org)... 140.211.166.134, 64.50.233.100, 64.50.236.52, ...
Connecting to packages.ros.org (packages.ros.org)|140.211.166.134|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 41968 (41K) [application/x-troff-man]
Saving to: 'python3-catkin-pkg-modules_0.4.20-1_all.deb'

python3-catkin-pkg-modules_0.4.20-1_all.deb          100%[=====================================================================================================================>]  40.98K  29.4KB/s    in 1.4s    

2020-05-18 11:38:47 (29.4 KB/s) - 'python3-catkin-pkg-modules_0.4.20-1_all.deb' saved [41968/41968]

Is this the correct state, or did I made any mistakes?

Originally posted by Kenji Miyake on ROS Answers with karma: 307 on 2020-05-18

Post score: 0

Original comments

Comment by gvdhoorn on 2020-05-18:
I've seen this as well, but for python3-vcstool (ros-industrial/industrial_ci#509 and ros-industrial-consortium/tesseract#292).

I can see that there are three addresses for packages.ros.org.

this could be an issue with round-robin DNS and not all hosts being in-sync.

Is this the correct state, or did I made any mistakes?

No, I believe your analysis is correct.

We'll have to wait for either:

  1. the hosts to get back in sync by themselves, or
  2. the admin(s) of the hosts to force them to sync

@tfoote @nuclearsandwich?

Edit: now also on ROS Discourse: comment on Preparing for Kinetic Sync 2020-05-13.

Comment by Kenji Miyake on 2020-05-18:
Thank you for your quick reply. I'll watch the discourse thread.

Could you tell me whether this is an accidental problem or could happen regularly?

this could be an issue with round-robin DNS and not all hosts being in-sync.

Do you mean that this kind of problem can frequently happen right after releasing a new version?
If so, do you know about how long we should wait for syncing?
Also, what is the best way to let admins know the problem when I find it? (As you did, replying to the Preparing thread?)

Comment by gvdhoorn on 2020-05-18:\

Could you tell me whether this is an accidental problem or could happen regularly?

This is the first time this has happened, IIRC.

Do you mean that this kind of problem can frequently happen right after releasing a new version?

No, that's not what I meant. What I wrote is that it seems there is a round-robin DNS configuration and in this particular case one of the hosts involved in that seems to have gone out-of-sync with the others.

If so, do you know about how long we should wait for syncing?

I would not know.

Also, what is the best way to let admins know the problem when I find it?

Posting either here or on ROS Discourse, maybe in the Packaging and Release Management category. Or perhaps the Buildfarm category.

Comment by gvdhoorn on 2020-05-18:
Seems it's being worked on: Incident Report for ROS: packages.ros.org mirrors out of sync.

Comment by Kenji Miyake on 2020-05-18:
I understand, thank you! I'll wait for a while and check again after it's fixed.

Comment by Kenji Miyake on 2020-05-18:
Oh, now 64.50.236.52 seems to be synced. Will check other packages as well for just in case.

Comment by Kenji Miyake on 2020-05-18:
I've tried the installation of ROS melodic + colcon several times using docker, and it seems working correctly.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Rosanswers logo

According to Incident Report for ROS: packages.ros.org mirrors out of sync, this should now be fixed:

Resolved

The mirrors are re-synchronized.

Original incident report:

We have identified that one of our 3 mirrors of packages.ros.org is out of sync. We're working with our hosting provider to bring the new one up to date.

Originally posted by gvdhoorn with karma: 86574 on 2020-05-19

This answer was ACCEPTED on the original site

Post score: 0

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.