I'm not quite old in quadcopter, but from my understanding, you can, but it is quite annoying.
We set up the PID control loops as position control loop and speed control loop because these are two ways we want and we will control the quadcopter.
For example, I want my quadcopter rotate 3 rad/s in yaw then stop or fly 5 meters away in y axis then stop. So I just set the reference signal as 3 in speed control loop and 5 in position control loop. That is enough.
But if we want to do the same thing with PID control for every axis, you need to calculate how these three inputs are and when to change them in order to make quadcopter stop. That is not quite convenient.