We can think of "jitter" as a fairly quick, fairly small perturbation upon the intended controlled behavior. A semi-quantitative design guideline (like "within 5%") needs a numerator and denominator. An arm that is supposed to move 20 cm allowing 1 cm wobble (5% of the distance) sounds pretty sloppy. A controller that overshoots by 5% might be acceptable if it settles down to an accurate, repeatable position. It's hard to be more specific with something so general.
Many things could present as jitter, with different causes, hence different ways to mitigate them. Here are a few.
A servo tries to move the load until the position sensor (feedback) corresponds to the command. Hobby servos, in particular, are tuned to manipulate some typical moment of inertia. A load with greater inertia will need more power to get it moving, and will overshoot. With large enough inertia, it can even oscillate without ever settling down. A useful phrase for googling is "PID control". To mitigate, there needs to be a better match between the load and the PID tuning, either by changing the design or by changing the tuning (if possible).
Suppose the drive chain has noticeable static friction. To start things moving, the controller has to send more and more power to the motor until the thing finally gets unstuck. But then there's more instantaneous power than is needed to maintain the motion, so the system overshoots and then overcorrects. This nonlinear phenomenon is different from underdamped servoing in a linear system, although it looks similar. Mitigation measures include lubrication and beefier but tighter control.
- system not stiff enough (bouncy)
Suppose the system is moving, and the motor abruptly stops. A system that isn't stiff enough will bounce around the endpoint. An example is in https://www.youtube.com/watch?v=HNpWsKflcMg at timestamp 3:51. (I had the sound off and skipped past the, uh, entertainer.) Other points, perhaps not simply bounce, are at 4:53 and 5:39. Mitigation measures include stiffening, reduce moment of inertia, and more sophisticated control.
- underpowered power supply (voltage sag)
Hobby servos, for one instance, misbehave in all sorts of creative ways when the motors draw more current from the power supply than it can comfortably deliver, leading to voltage sag. Mitigation: have more current available, or smaller motors.
- electrical noise, poor grounding
Noise in a sensor or command signal will effectively lie to the controller about what it's supposed to do, or corrupt the feedback as to what it's actually doing. The controller can never settle down, instead constantly doing the herky-jerky. I can't be too specific about mitigation, because there are so many ways things can go wrong electrically. Try not to have motor current sharing conductors with small signals, and otherwise carefully review grounding. Capacitors and shielding help sometimes.