I was going to add this as a comment, but it's more detailed than one so I'll post it as an answer.
You might be able to measure to the sub-millimeter level, but you will find it very, very difficult to control to that level. This is because every mechanical structure has some form of resonance - everything shakes.
If your robot were just a chunk of aluminum, you'd probably be fine. But, because you want something functional, you put joints in the robot. Those joints need to be flexible to be functional, but they also need to be rigid enough (when commanded) to support the weight of the arm plus whatever payload you have.
The fact that a robot moves to a commanded position means that there will be some momentum in the arm. When you bring the arm to a stop there is a jerk that mechanically excites the arm, starting vibration. You can measure the motion of the arm, but controlling the vibration with an actuator, to the level you're talking about, is very difficult.
Regarding measurement accuracy, you haven't given any dimensions for your arm, so it's not possible to give a hard spec on your encoder resolution requirement, but I'll wager that what you have is insufficient.
If your arm is 1 foot long (305mm), then motion of 0.1mm at the end of that arm is equivalent to
$dx = r\sin{(\theta)}$
$\theta = \mbox{asin}{(dx/r)}$
$\theta = 0.018 \mbox{deg}$
You state that your encoder measures to 0.03% of 360 degrees, which means is measures to 30% of 0.36 degrees, or about 0.1 degree. This means that you need 10 times the resolution on your encoder if your arm, from center of the joint to the end of the end effector, is ~300 mm.
Whenever you do get data, you can quickly check for resonance by plotting the autocorrelation of data. Resonance generally some sinusoidal function; this would be pretty apparent in the plot.
If you get an encoder that has exactly the resolution you think you need, then sometimes you can get "noise" where the value being measured is on the edge of a bit threshold. For this reason, I typically try to get an encoder a couple bits higher than what I think I need. I thought there was a thumb rule for this but I can't find it in a cursory search online and I don't have my measurement text with me.
Hopefully this gives you some guidance; if you can update your question (please don't post as a comment) a link to your data along with the dimensions of your arm then it would be possible to give you a more thorough answer.