I've been reading about ORB-SLAM and ORB-SLAM2, but I can't quite figure out how pose estimation is done. I see that it uses a bag-of-word method for loop detection, but how are current features matched against the features of the map? As the map is 3D, a sought change in position can cause different perspectives. Bag-of-word can't really address the problem of perspectives, right?
Bag-of-word is just a similarity check routine. Perspective change is more related to the robustness of the feature descriptor against view change. Also, the map is 3D but I guess they are using 2D image-based place recognition only.
"The vocabulary is created offline with the ORB descriptors extracted from a large set of images." section III.E of ORB-paper.
So, I guess given a new image they extract orb features and compare them to the orb features in the DB.