The example you have given is not self-motion - and I believe you understand why already. Even though the end-effector does not change position, it does change orientation. Mathematically speaking, this means the output of our end-effector forward kinematics function has changed with our most recent motion. So - we can simply define null-motion (self-motion) as any joint velocity which once applied to our robot's joints do not change the output of our forward kinematics. This is equivalent to the following constraint:
$$ \dot{\mathbf{x}} = \mathbf{J}(\mathbf{q})\dot{\mathbf{q}} = 0 $$
where $\dot{\mathbf{x}}$ is the end-effector task space velocity, $\mathbf{q}$ is the current joint configuration, $\dot{\mathbf{q}}$ is the current joint velocity, and $\mathbf{J}(\cdot)$ is the geometric Jacobian. Now, what aspects of the forward kinematics function you care about are up to you. If you do not care about orientation, one might relax null-motion to the following constraint:
$$ \dot{\mathbf{x}} = \mathbf{J}(\mathbf{q})\dot{\mathbf{q}} = \begin{bmatrix} 0 \\ \omega \end{bmatrix} $$
where $\omega$ is some arbitrary angular velocity.
Some notes:
- If the number of joints your robot has is equal to the number of task space degree's of freedom, null-motion is only achievable at singular configurations.
- Given $n$ joints, $m$ task space DoFs, and $n \gt m$ - The null space or null space projection matrix can be solved as $(I - \mathbf{J}^{\dagger}\mathbf{J})$, where $\mathbf{J}^{\dagger}$ is the pseudo-inverse of the geometric Jacobian. Any $\dot{\mathbf{q}}$ such that $\dot{\mathbf{q}} = (I - \mathbf{J}^{\dagger}\mathbf{J})\dot{\mathbf{r}}$, where $\dot{\mathbf{q}}, \dot{\mathbf{r}} \in \mathbb{R}^{n}$, will result in no end-effector motion. see here.
- When $n \gt m$, null-motion lies on what are referred to as self-motion manifolds. These manifolds are typically of $n-m$ dimension and act as continuous solutions to the inverse kinematics problem. This also means that we can continuously move through our null-space without discontinuity - although for the case of revolute joints we will return to our original starting configuration eventually.
