This problem can be made easier when formulating an equivalent problem using unit quaternions
$$
q\,q_a = q_b\,q \tag{1}
$$
where each quaternions can be expressed as $w + x\,i + y\,j + z\,k$ while satisfying that $w^2 + x^2 + y^2 + z^2 = 1$, where $i^2 = j^2 = k^2 = i\,j\,k = -1$ (it can be noted that $i$, $j$ and $k$ do not commute, for example $i\,j \neq j\,i$).
For quaternions multiplication it can be shown that it can be written as a matrix vector product, where the matrix is a function of one of the two quaternions. Namely when considering the quaternions $q = q_1 + q_2\,i + q_3\,j + q_4\,k$ and $r = r_1 + r_2\,i + r_3\,j + r_4\,k$, then their product can be written as
$$
\begin{align}
q\,r &=
\begin{bmatrix}
1 & i & j & k
\end{bmatrix}
\begin{bmatrix}
q_1 & -q_2 & -q_3 & -q_4 \\
q_2 & q_1 & -q_4 & q_3 \\
q_3 & q_4 & q_1 & -q_2 \\
q_4 & -q_3 & q_2 & q_1
\end{bmatrix}
\begin{bmatrix}
r_1 \\ r_2 \\ r_3 \\ r_4
\end{bmatrix} \\
&=
\begin{bmatrix}
1 & i & j & k
\end{bmatrix}
\begin{bmatrix}
r_1 & -r_2 & -r_3 & -r_4 \\
r_2 & r_1 & r_4 & -r_3 \\
r_3 & -r_4 & r_1 & r_2 \\
r_4 & r_3 & -r_2 & r_1
\end{bmatrix}
\begin{bmatrix}
q_1 \\ q_2 \\ q_3 \\ q_4
\end{bmatrix}
\end{align}
$$
Dividing by a unit quaternions is the same as multiplication with its conjugate, which is defined as negating the complex parts: $w - x\,i - y\,j - z\,k$. This allows us to write equation $(1)$ as
$$
q\,q_a\,q^{-1} = q_b
$$
which is equivalent to
$$
\begin{bmatrix}
q_1 & -q_2 & -q_3 & -q_4 \\
q_2 & q_1 & -q_4 & q_3 \\
q_3 & q_4 & q_1 & -q_2 \\
q_4 & -q_3 & q_2 & q_1
\end{bmatrix}
\begin{bmatrix}
q_1 & q_2 & q_3 & q_4 \\
-q_2 & q_1 & -q_4 & q_3 \\
-q_3 & q_4 & q_1 & -q_2 \\
-q_4 & -q_3 & q_2 & q_1
\end{bmatrix}
\begin{bmatrix}
q_{a,1} \\ q_{a,2} \\ q_{a,3} \\ q_{a,4}
\end{bmatrix} =
\begin{bmatrix}
q_{b,1} \\ q_{b,2} \\ q_{b,3} \\ q_{b,4}
\end{bmatrix}. \tag{2}
$$
It can be shown that, while using the fact that $q$ is a unit quaternion, the two matrices in the above equation can be combined into the following expression
$$
\begin{bmatrix}
q_1 & -q_2 & -q_3 & -q_4 \\
q_2 & q_1 & -q_4 & q_3 \\
q_3 & q_4 & q_1 & -q_2 \\
q_4 & -q_3 & q_2 & q_1
\end{bmatrix}
\begin{bmatrix}
q_1 & q_2 & q_3 & q_4 \\
-q_2 & q_1 & -q_4 & q_3 \\
-q_3 & q_4 & q_1 & -q_2 \\
-q_4 & -q_3 & q_2 & q_1
\end{bmatrix} =
\begin{bmatrix}
1 & 0 & 0 & 0 \\
0 & 1-2(q_3^2+q_4^2) & 2(q_2q_3-q_1q_4) & 2(q_2q_4+q_1q_3) \\
0 & 2(q_2q_3+q_1q_4) & 1-2(q_2^2+q_4^2) & 2(q_3q_4-q_1q_2) \\
0 & 2(q_2q_4-q_1q_3) & 2(q_3q_4+q_1q_2) & 1-2(q_2^2+q_3^2)
\end{bmatrix} \tag{3}
$$
the lower right $3\times3$ matrix of the the right hand side of $(3)$ is just the rotation matrix representation of $q$, denoted as $R(q)$. Substituting this result back into equation $(2)$ then we can conclude that there can only be a solution for $q$ given $q_a$ and $q_b$ if and only if $q_{a,1} = q_{b,1}$. This essentially comes down to that both $q_a$ and $q_b$ represent a rotation of the same angle, which would be identical to the requirement that the rotation matrices $R_1$ and $R_2$ have the same eigenvalues/are similar. Another way of writing a quaternion is the axis-angle representation $\cos\left(\theta\over2\right) + \sin\left(\theta\over2\right)(u_x\,i+u_y\,j+u_z\,k)$, where $\theta$ is the angle and $\begin{bmatrix}u_x & u_y & u_z\end{bmatrix}^\top$ the (unit-)axis of rotation. So therefore we can conclude from equation $(2)$ and $(3)$ that $R(q)$ has to map the rotation axis of $q_a$, $\vec{u}_a$, to the rotation axis of $q_b$, $\vec{u}_b$, so $\vec{u}_b = R(q)\,\vec{u}_a$. There are infinitely many rotations which would satisfy this mapping.
The rotations with the smallest angle can be found by using a rotation axis perpendicular to both given axes, which can be obtained using the cross product of the two axes
$$
q_{\min} = \sqrt{\frac{1 + \vec{u}_a \cdot \vec{u}_b}{2}} + \sqrt{\frac{1 - \vec{u}_a \cdot \vec{u}_b}{2}}
\begin{bmatrix}
i & j & k
\end{bmatrix}
\frac{\vec{u}_a \times \vec{u}_b}{\|\vec{u}_a \times \vec{u}_b\|} \tag{4}
$$
where $\vec{u}_a$ and $\vec{u}_b$ are both assumed to be of unit length.
The rotations with the largest angle can be found by rotating 180° around the average of the two axis
$$
q_{\max} =
\begin{bmatrix}
i & j & k
\end{bmatrix}
\frac{\vec{u}_a + \vec{u}_b}{\|\vec{u}_a + \vec{u}_b\|}. \tag{5}
$$
And if it is desired these could be converted to rotation matrices. For example the rotation matrix corresponding to equation $(5)$ can simplified down to the following expression
$$
R_{\max} = \frac{1}{s_1^2 + s_2^2 + s_3^2}
\begin{bmatrix}
s_1^2-s_2^2-s_3^2 & 2\,s_1\,s_2 & 2\,s_1\,s_3 \\
2\,s_2\,s_1 & s_2^2-s_1^2-s_3^2 & 2\,s_2\,s_3 \\
2\,s_3\,s_1 & 2\,s_3\,s_2 & s_3^2-s_1^2-s_2^2
\end{bmatrix} \tag{6}
$$
with $s_n = q_{a,n+1} + q_{b,n+1}\ \forall\,n\in\{1,2,3\}$.
A = kron(I, R2) - kron(R1', I);
and after this you can also get the solutions by looking at the null space of A:null(A)
. But from my testing it seems that you will always get three solutions, so any linear combination of those is still a valid solution. So you would still need to find which linear combination yields a rotation matrix. $\endgroup$