# Maintaining scale factor for essential matrix

I want to recover the trajectory of a vehicle using a monocular camera via the computation of the essential matrix between t-1 and t. When I used OpenCV, I got correct trajectory wrt the ground truth. However, I want to code all the functions in Matlab but I got garbage when I plotted the trajectory and I think it is related to a scale factor problem.

In fact, the outputted essential matrix from opencv function is the following (between two consecutive frames)

$$E = \begin{bmatrix} 0.0052 & -0.7068 & 0.0104\\ 0.0052 & -0.7068 & 0.0104\\ 0.7063 & 0.0050 & -0.0305\\ -0.0113 & 0.0168 & 0.0002\\ \end{bmatrix}$$

After decomposing it into Rotation and translation and triangulating for 5 2D points, I got the following 3D points:

$$X1 =\begin{bmatrix} -0.0940& 0.0478& -0.4984\\ -0.0963& 0.0497& -0.4987\\ 0.3033& 0.1009& -0.5202\\ -0.0065& 0.0636& -0.5053\\ -0.0737& 0.0653& -0.5011\\ \end{bmatrix}$$

Now, for the essential matrix outputted by Matlab functions, it is the following:

$$E2=\begin{bmatrix} -0.2153 & 0.9573 & 0.1626\\ 0.8948 & 0.2456 &-0.3474\\ 0.1003 & 0.1348 &-0.0306\\ \end{bmatrix}$$

When decomposing it into rotation and translation as well, and triangulating points, I got the following 3D points:

$$X2 =\begin{bmatrix} 0.1087& -0.0552& 0.5762\\ 0.1129& -0.0578& 0.5836\\ 0.4782& 0.1582& -0.8198\\ 0.0028& -0.0264& 0.2099\\ 0.0716& -0.0633& 0.4862\\ \end{bmatrix}$$

To verify the results, I did:

$$X1./X2 =\begin{bmatrix} -0.8644 & -0.8667 & -0.8650\\ -0.8524 & -0.8603 & -0.8546\\ 0.6343 & 0.6376 & 0.6346\\ -2.3703 & -2.4065 & -2.4073\\ -1.0288 & -1.0320 & -1.0305\\ \end{bmatrix}$$

And an almost constant scale factor seems to exist between the first and second estimations. I think that the scale factor have to be the same for all 3D points to get a correct trajectory when plotting it. How to maintain the scale factor?