I have been reading about scan matching and so far I have gathered that scan matching tries to find the transformation (rotation and translation) which leads to the best match between overlapping points of the two scans. My question is : what algorithms does it use to find the overlapping regions ? Any explanations or links to relevant papers are greatly appreciated.
The process you are referring to is called point cloud registration (or point matching). The goal of point cloud registration is find the spatial transformation that aligns two point clouds (i.e., sets of points). One of the most popular methods is iterative closest point (ICP), and many variants of ICP exist. Other methods exist as well such as robust point matching (RPM) and variants, but I am not familiar with RPM.
The basic ICP algorithm essentially works by matching closest points and minimizing the error between point matches to find the rotation and translation between point clouds. The process is then repeated after transforming the original point cloud.
Methods for ICP are provided in the Point Cloud Library (PCL), which is open source software for point cloud processing. Other software also includes implementation e.g., MATLAB, libpointmatcher. If you are interested in the theory, you can see the original ICP paper:
P. Besl, N. McKay "Method for Registration of 3-D Shapes" in IEEE Transactions of Pattern Analysis and Machine Intelligence, 1992.