Consider an arbitrary rotation matrix $\bf R$ $\in$ SO(3). We can use it transform a vector $v \in {\mathbb R}^3$ by $$ w = {\bf R}v$$
Intuitively, if $v$ corresponds to the rotation axis of $\bf R$ then it will be unchanged by the rotation, that is, $w = v$.
So the rotation axis must be an eigenvector of $\bf R$. Since $\bf R$ is a 3x3 matrix it has 3 eigenvectors. Each eigenvector has an associated eigenvalue: there is always at least one eigenvalue equal to 1 (ie. the vector's length is unchanged by the rotation), and for non-zero rotation the other two eigenvalues are a complex conjugate pair. The eigenvector corresponding to the eigenvalue equal to one is the rotation axis.
A simple example in MATLAB
>> R = roty(0.3)
R =
0.9553 0 0.2955
0 1.0000 0
-0.2955 0 0.9553
which is a rotation of 0.3 radians about the y-axis (this function is from my Robotics Toolbox for MATLAB).
>> [x,e] = eig(R)
x =
0.7071 + 0.0000i 0.7071 + 0.0000i 0.0000 + 0.0000i
0.0000 + 0.0000i 0.0000 + 0.0000i 1.0000 + 0.0000i
0.0000 + 0.7071i 0.0000 - 0.7071i 0.0000 + 0.0000i
e =
0.9553 + 0.2955i 0.0000 + 0.0000i 0.0000 + 0.0000i
0.0000 + 0.0000i 0.9553 - 0.2955i 0.0000 + 0.0000i
0.0000 + 0.0000i 0.0000 + 0.0000i 1.0000 + 0.0000i
where x
is a matrix whose columns are the eigenvectors, and e
is a diagonal matrix of eigenvalues. The unit eigenvalue is in column 3 so our rotation axis is the third column of x
which is real and equal to $[0, 1, 0]^T$ which is a unit-vector parallel to the y-axis — the original axis of rotation.
The complex eigenvalues tell you about the amount of rotation around the axis
>> angle(e(1,1))
ans =
0.3000
The Toolbox has a function which does all this for you
>> [theta,v] = tr2angvec(R)
theta =
0.3000
v =
0 1.0000 0
To go in the opposite direction, from a rotation axis and angle, to a rotation matrix use Rodrigues' formula. This is provided in the Toolbox as
>> angvec2r(theta,v)
ans =
0.9553 0 0.2955
0 1.0000 0
-0.2955 0 0.9553