# Simulated raytracing in rviz or gazebo

Hello,

I have an application where I have a camera attached to the end of a 5-DOF robot arm (X, Y, Z, yaw, pitch). The robot arm is moving the camera around a fixed object (0,0,0) for inspection purposes. Given the position and orientation of the camera, is it possible to detect where on the object the camera is seeing? For example, I would like to know whether the axis of the camera is intersecting the object, and if so, determine the exact coordinates of the object the camera axis is intersecting.

Given the known camera axis (position/orientation), is it possible to extend a line along the +x-axis and see where it hits the fixed object (if it all)?

I should be able to calculate positions along the +x-axis moving outwards from the camera frame. However, I do not know how to check if these points are contained inside of the fixed object of interest. Would also be nice if something already existed for this, or at least something I can use to help.

Thanks

Originally posted by JoshMarino on ROS Answers with karma: 592 on 2017-07-20

Post score: 2

Comment by Airuno2L on 2017-07-21:
Are you sensing the object somehow? If so, how?

Comment by JoshMarino on 2017-07-21:
I know what the object is beforehand, and it is placed at world coordinates (0,0,0).

Comment by Airuno2L on 2017-07-21:
I'm a little confused, if you know it is at (0,0,0) why do you do say "determine the world coordinates of the object the camera is looking at"? Aren't the world coordinates of the object are going to be (0,0,0)?

Comment by JoshMarino on 2017-07-21:
Sorry, guess I was not clear. The object is very large wrt the camera. I would like to know where on the object I am looking at through the center of the camera, not the center of the object (0,0,0).

Comment by intee on 2021-08-04:
@Airuno2L, could you please have a look at this query? thanks

Although I'm not 100% sure what you're trying to do, the image_geometry library might have some useful stuff in it.

For example, the project3dToPixel(point) and/or projectPixelTo3dRay(uv) might be useful.

If you're wondering if, at any given time, the camera's center axis is intersecting the objects bounding box, you might have to implement that by hand, or grab an example from online somewhere by searching for something like "code for ray passing through a bounding box".

Or are you trying to find the world coordinates of the point at which the camera's center axis intersects the object? If so, is the object shape arbitrary? Or is it like a cube or sphere or some other geometric primitive shape?

EDIT: I understand better now. What I would do is convert my object model to an octree and do ray tracing on that. There are two options that I know of, one is to use Octomap which has the castRay function or use the Octrees in Point Cloud Library which have the OctreePointCloudSearch functions.

In either case you'll need to convert your model into something that one of those libraries can use. Depending on what format the object is already in, here are some hints for doing it with Octomap. For Point Cloud Library, I know you can at least import from a PLY file using this class.

Then just use the camera location as the origin of the ray and the camera direction as the ray direction and bingo! For what it's worth, I think I've used PCL and Octomap to do something similar in the past and don't recall either one being easier or harder than the other, so I don't think it matters too much which you choose.

Originally posted by Airuno2L with karma: 3460 on 2017-07-21

This answer was ACCEPTED on the original site

Post score: 3