# How to estimate relative scale correctly in a Monocular SFM system?

I am quite new to this whole SFM and VO stuff and I am sometimes really impressed what is possible.

Therefore I decided to do my own very basic project, toget in touch with this stuff. Everything seems to work so far. However now I need some help, because I just do not get it. Even when I looked through code of SFM, I cannot see how its done.

Imagine I use a monocular system and capture a video of about 600 frames with 30 fps. I already got the Essential matrix for each subsequent pair of frames, I estimated the Camera matrix using auto-calibration and I calculated rotation matrix R and transaltion matrix t for every subsequent frame pair.

Now I want to calculate the relative scale and this is where I struggle. I understand the basic idea:

Take 3 frames, match feature points over all 3 frames, calculate the corresponding 3D points and calculate relative scale by pairing 3D points and applying this formular: However I am highly confused what are my P matrices. In more detail, I am stuck at this kind of code:

    3d_points_Frame1and2=triangulatePoints(P_FromFrame1To2, P_FromFrame1To2, p1, p2);
3d_points_Frame2and3=triangulatePoints(P_FromFrame2To3, P_FromFrame2To3, p2, p3);


Where triangulatePoints, calculates the 3D points using the camera matrices P1 and P2 and the points given as p1, p2. Is the idea correct? Or do I need different P matrcies? Can I use two times the same P matrix? It is basically the same camera, so it should work, shouldn't it?

And here actually starts my problem: What exactly are my P matrices? How cann I calculate them?

I know that, P is calculatable by using R and t. There is no problem to calculate P individually for each pair of frames. However how do I use them to recover scale? I am quite sure, that I need an accumulated P matrix when I want to recover scale, dont I?

Sorry for the confused writing. But I am not even exactly sure, what I want to know... :/ I just want to recover relative scale in a monocular system using already calculated R and t matrices, where t ist an unit vector.

I hope you still can help me, even though I am a bit confused.