There's a general procedure for estimating projection transformation between camera and 3d lidar but there's a secondary problem which is associating pixels with depths in a grid which isn't uniformly sampled.
What is the correct approach to doing this? Both the pre and post transformed depth points lack a grid structure. The pre-transformed points lack a grid structure because the sampling is time instead of angle based so you get variations in the angles. The post-transformed grid is obviously not going to be uniformly sampled either,
