Re-implementing your solution, I get this:
Angle Between Vectors
First, you want the angle between points $A$ and $B$ -- not specifically the unit vector.
(via Fx Programming):
$\theta = math.atan2(B_{x}-A_{x}, B_{y}-A_{y})$
Vehicle Yaw Angle
Next (and I suspect this is your problem), you need to subtract the vehicle's yaw angle $\psi$ from your calculated $\theta$.
Heading vs Yaw
If you're using a compass for the "yaw angle" of your vehicle, this could also be your mistake; heading and yaw are not the same.
Compass heading is zero along the positive $y$ axis, increasing as it turns clockwise:
Yaw is zero along the positive $x$ axis, increasing as it turns counterclockwise:
The 90 degree overlap between these measurements, combined with adding (instead of subtracting) the vehicle yaw from the desired yaw, may be why things worked when your target was within ±5° and behaved badly at ±90°.
Conversion to Component X and Y
From there, you say that you are converting this result $(\theta-\psi)$ into its $x$ and $y$ components, passing them to the robot as the roll and pitch angles. With the above corrections, you should get the desired result at this point. However, directly mapping these components to tilt angles might be problematic since you are only considering the difference in position, and not the velocity (really, the momentum) of the vehicle.
PID Control
You may be best served by using PID control loops for the roll and pitch of the vehicle. That is, once you fix your code and are able to hit your target, my guess is that you'll start overshooting it instead -- oscillating back and forth. A correctly-tuned PID will prevent that from happening while still allowing you to approach the target quickly.
Instead of plugging your $x$ and $y$ into roll and pitch, consider them to be the error values that the roll and pitch PIDs accept as input.