0
$\begingroup$

Rosanswers logo

I am unable to build autoware with GPU support. I just downgraded CUDA to 10.0 and tried to built by using CUDA Support.

Operating System: Ubuntu 18.04 ROS: Melodic GCC version: GCC (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0 CMake version : 3.10.2 ROS distribution: melodic ROS installation type: sudo apt-get install ros-melodic-desktop-full GPU MODEL: GeForce GTX 1060M CUDA Version: 10 CUDNN Version: 7.5 Autoware Installation type : Built from source (https://gitlab.com/autowarefoundation/autoware.ai/autoware/wikis/Source-Build) Autoware Version : 1.12.0 EIgen Version : 3.3.7

:~$ nvidia-smi
Sun Aug 11 19:09:22 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 415.27       Driver Version: 415.27       CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1060    Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   68C    P0    28W /  N/A |    558MiB /  6078MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0       505      G   ...uest-channel-token=17405615527381595598   112MiB |
|    0     22094      G   /usr/lib/xorg/Xorg                           309MiB |
|    0     22229      G   /usr/bin/gnome-shell                         133MiB |
+-----------------------------------------------------------------------------+
~$ pkg-config --modversion eigen3
3.3.7

I am getting following error when I am building using cuda support.

:~/autoware.ai$ AUTOWARE_COMPILE_WITH_CUDA=1 colcon build --cmake-args -DCMAKE_BUILD_TYPE=Release
Starting >>> autoware_build_flags
Starting >>> autoware_msgs
Starting >>> vector_map_msgs
Starting >>> autoware_config_msgs
Starting >>> tablet_socket_msgs
Starting >>> autoware_system_msgs
Starting >>> autoware_can_msgs
Starting >>> gnss
Finished <<< autoware_build_flags [2.01s]                                                                                                                   
Starting >>> sick_ldmrs_description
Finished <<< gnss [2.43s]                                                                                                                           
Starting >>> sick_ldmrs_msgs
Finished <<< sick_ldmrs_description [0.75s]                                                                                                                     
Starting >>> ds4_msgs
Finished <<< autoware_can_msgs [3.27s]                                                                                                                                 
Starting >>> custom_msgs
Finished <<< tablet_socket_msgs [3.57s]                                                                                                               
Starting >>> ds4_driver
Finished <<< autoware_system_msgs [3.81s]                                                                                                             
Starting >>> autoware_health_checker
Finished <<< ds4_msgs [1.57s]                                                                                                                         
Starting >>> gazebo_camera_description
Finished <<< ds4_driver [0.86s]                                                                                                                       
Starting >>> gazebo_imu_description
Finished <<< sick_ldmrs_msgs [2.09s]                                                                                                                  
Starting >>> sick_ldmrs_driver
Finished <<< gazebo_camera_description [1.11s]                                                                                                        
Starting >>> ndt_cpu
Finished <<< gazebo_imu_description [1.11s]                                                                                                            
Starting >>> ndt_tku
Finished <<< ndt_cpu [0.67s]                                                                                                                           
Starting >>> pcl_omp_registration
Finished <<< ndt_tku [0.64s]                                                                                                                           
Starting >>> fastvirtualscan
Finished <<< autoware_config_msgs [6.91s]                                                                                                                
Starting >>> kitti_player
Finished <<< pcl_omp_registration [0.83s]
Starting >>> ndt_gpu
Finished <<< custom_msgs [3.70s]                                                                                                                
Starting >>> rosinterface
Finished <<< fastvirtualscan [0.78s]
Starting >>> autoware_bag_tools
Finished <<< rosinterface [1.31s]                                                                                                                         
Starting >>> autoware_external_msgs
--- stderr: ndt_gpu                                                                                                                                       
** WARNING ** io features related to ensenso will be disabled
** WARNING ** io features related to davidSDK will be disabled
** WARNING ** io features related to dssdk will be disabled
** WARNING ** io features related to pcap will be disabled
** WARNING ** io features related to png will be disabled
** WARNING ** io features related to libusb-1.0 will be disabled
** WARNING ** visualization features related to ensenso will be disabled
** WARNING ** visualization features related to davidSDK will be disabled
** WARNING ** visualization features related to dssdk will be disabled
** WARNING ** visualization features related to rssdk will be disabled
CMake Error at /home/sai/autoware.ai/install/autoware_build_flags/share/autoware_build_flags/cmake/autoware_build_flags-extras.cmake:32 (message):
  GPU support on Melodic requires Eigen version>= 3.3.7
Call Stack (most recent call first):
  CMakeLists.txt:22 (AW_CHECK_CUDA)


---
Failed   <<< ndt_gpu    [ Exited with code 1 ]
Aborted  <<< autoware_bag_tools                                                                                                                                        
Aborted  <<< sick_ldmrs_driver                                                                                                                                         
Aborted  <<< autoware_msgs                                                                                                                                              
Aborted  <<< vector_map_msgs                                                                                                                                           
Aborted  <<< autoware_external_msgs                                                                                                                                
Aborted  <<< kitti_player                                                                                                    
Aborted  <<< autoware_health_checker                                             

Summary: 18 packages finished [12.0s]
  1 package failed: ndt_gpu
  7 packages aborted: autoware_bag_tools autoware_external_msgs autoware_health_checker autoware_msgs kitti_player sick_ldmrs_driver vector_map_msgs
  1 package had stderr output: ndt_gpu
  115 packages not processed

Originally posted by AM97 on ROS Answers with karma: 139 on 2019-08-11

Post score: 0


Original comments

Comment by Josh Whitley on 2019-08-12:
I thought we might start to run into CUDA compatability problems soon. Can you please run the following commands and provide the output?

./opt/cuda/bin/nvcc --version
./usr/local/cuda/bin/nvcc --version
uname -r
lsb_release -a
apt search cuda | grep installed

Comment by AM97 on 2019-08-13:
@Maximus5684 uname -r 4.15.0-55-generic lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 18.04.3 LTS Release: 18.04 Codename: bionic /usr/local/cuda/bin$ nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2017 NVIDIA Corporation Built on Fri_Nov__3_21:07:56_CDT_2017 Cuda compilation tools, release 9.1, V9.1.85

Comment by AM97 on 2019-08-13:
@Maximus5684 apt search cuda | grep installed

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

cuda-command-line-tools-10-0/unknown,now 10.0.130-1 amd64 [installed,auto-removable] cuda-compiler-10-0/unknown,now 10.0.130-1 amd64 [installed,auto-removable] cuda-cublas-10-0/unknown,now 10.0.130-1 amd64 [installed,auto-removable] cuda-cublas-dev-10-0/unknown,now 10.0.130-1 amd64 [installed,auto-removable] cuda-cudart-10-0/unknown,now 10.0.130-1 amd64 [installed,auto-removable] cuda-cudart-dev-10-0/unknown,now 10.0.130-1 amd64 [installed,auto-removable] cuda-cufft-10-0/unknown,now 10.0.130-1 amd64 [installed,auto-removable] cuda-cufft-dev-10-0/unknown,now 10.0.130-1 amd64 [installed,auto-removable] cuda-cuobjdump-10-0/unknown,now 10.0.130-1 amd64 [installed,auto-removable] cuda-cupti-10-0/unknown,now 10.0.130-1 amd64 [installed,auto-removable] cuda-curand-10-0/unknown,now 10.0.130-1 amd64 [installed,auto-removable] cuda-curand-dev-10-0/unknown,now 10.0.130-1

Comment by AM97 on 2019-08-13:
@Maximus5684 I changed so that no mismatches are there nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2018 NVIDIA Corporation Built on Sat_Aug_25_21:08:01_CDT_2018 Cuda compilation tools, release 10.0, V10.0.130

but still I am getting same problem!!!!!!

Comment by Josh Whitley on 2019-08-14:
@AM97 - Please also give me the output of uname -r. I need to know which kernel version you are running. Also, make sure that there are not multiple versions of CUDA installed. You can check to see if there are multiple NVCCs by doing:

sudo updatedb
sudo locate nvcc

Comment by AM97 on 2019-08-14:
@Maximus5684 uname -r 4.15.0-55-generic and the output of sudo locate nvcc is in this gistfile (https://gist.github.com/sandeepmaddula/82244ff22eaa003833b8c26b56824c57)

Comment by Josh Whitley on 2019-08-14:
The version in /usr/bin/nvcc is a problem - this file belongs to the Ubuntu package nvidia-cuda-toolkit which is an old version of CUDA (9.1). Please run the following command:

sudo apt purge nvidia-cuda-*
sudo apt autoremove

and then clean (delete build/, install/, and log/) your Autoware workspace, and re-build using

AUTOWARE_COMPILE_WITH_CUDA=1 colcon build --cmake-args -DCMAKE_BUILD_TYPE=Debug
$\endgroup$

1 Answer 1

0
$\begingroup$

Rosanswers logo

@AM97 You are currently using the version of CUDA that is provided by Ubuntu. On Ubuntu 18.04, you MUST install version 10.0 directly from Nvidia. To make sure that you do not have the version from Ubuntu, do the following:

sudo apt purge nvidia-cuda*
sudo apt autoremove

Then, to install 10.0, carefully follow the instructions at https://docs.nvidia.com/cuda/archive/10.0/cuda-installation-guide-linux/index.html. Do not install the nvidia-cuda-toolkit or nvidia-cuda-dev packages from the Ubuntu repositories - these install CUDA 9.1, which is not compatible with Autoware on Ubuntu 18.04.

In addition, on Ubuntu 18.04, you must upgrade the version of Eigen that is provided with Ubuntu by following the instructions at https://gitlab.com/autowarefoundation/autoware.ai/autoware/wikis/Source-Build#additional-system-dependencies-for-cuda-support.

After the above is done, you should be able to build with AUTOWARE_COMPILE_WITH_CUDA=1 to enable CUDA capabilities.


Originally posted by Josh Whitley with karma: 1766 on 2019-08-14

This answer was ACCEPTED on the original site

Post score: 1


Original comments

Comment by AM97 on 2019-08-14:
@Maximus5684 Thanks for the help. After some time I found it now I installed directly from NVidia. Now it was built successfully. Thanks a lot for your quick help.

Comment by Martin Peris on 2019-08-15:
It is just a matter of time until NVidia breaks everything again xD https://www.youtube.com/watch?v=_36yNWw_07g

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.