Google's 'Tango' - How it works and what's special about the hardware?

Question

I'm looking for a good breakdown and explanation of Google's 'Tango' AR platform. Specifically how the hardware works together to generate depth maps and the SDK's use of it.

I know the hardware composes of a fisheye lens camera and an RGB-I camera. I am only familiar with stereo vision with identical cameras and disparity maps, I am thinking the different lenses and camera elements make it easier to distinguish variations in the environment but must have some very special (and proprietary) algorithms. However, there must actually some special hardware and dedicated chipsets for processing the depth map to take the burden off the CPU/GPU?

Also, for the AR software implementation, I assume the SDK has some GPU utilization built into it like OpenCL or CUDA (but specific for the Adreno GPU). Does it simply use OpenCL (this is supported by the Adreno GPU) or does it have something proprietary from Google similar to CUDA for the nVidia chipsets?

Basis for the question - I work with OpenCV some and am experimenting with stereo vision applications, but would like to move on to developing apps for specialized hardware and this sounds like the right (maybe only?) platform.

Answer 1

I suspect, but don't know, that the Tango uses a variant of volumetric reconstruction using a Truncated Signed Distance Function (Good intro at http://www.cs.nyu.edu/courses/fall12/CSCI-GA.2945-001/dl/jiakai-slides.pdf)

It uses structured IR light to obtain a dense depth map, projects these points back into 3-space as a point cloud, probably turns this into a mesh by triangulating it and then projects it into a TSDF volume representing the space being mapped.

Marching Cubes or similar algorithm can then be used to extract the 0 crossing of this isosurface into a mesh.

Given that the Tango seems to bee able to map quite large areas, I would assume that either the mapping resolution is quite coarse or else Google have implemented a streaming algorithm to move the volumetric reconstruction into and out of GPU memory.

The seminal paper here is KinectFusion (https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/ismar2011.pdf) which describes how this was done with a Kinect using depth only data.