I am not a UAV guy, so others may be able to provide more specific answers. But from a controls perspective, I would break the problem into two separate areas: aerodynamics and controller response.
For aero, you want to consider both the inherent stability of your system, and the flight dynamics. Essentially you want the uncontrolled system to be as stable as possible, then add the controller to handle unwanted disturbances. For example, where is the cg of your robot? Would moving some masses lower lead to a more naturally stable system? You want as much inherent stability in the mechanics as you can get, so your controller can use its energy to combat (reject) the disturbances. How else can you improve its aerodynamics? What about a larger propeller, or more prop blades? Or stronger motors? Or a lighter frame? Getting a higher lift to weight ratio is a benefit to stability. Also, consider the system's cross-section if wind is contributing to the unsteadiness. A tapered profile may help prevent impulsive disturbances in windy conditions. These areas are complicated and beyond my experience. But hopefully they will help you look into the system's natural dynamics to see if improvements can be made there.
For the control responsiveness, there are also many items that may help. Of course, faster control loop rates, and highly accurate sensors help. What is the loop rate of your controller? Can you program it more efficiently? Or use a faster micro? And, is the most current sensor data getting to the controller quickly? I suspect a loop rate of around 1 kHz, and sensors with that type of bandwidth, would be required to achieve the type of stability you desire. I would also look at the responsiveness and accuracy of your altitude and attitude sensors. If you are using an IMU, could you add other sensors to get more accurate and/or faster data? What about a downward-pointing (possibly laser) rangefinder to get very peppy altitude measurements? Does your power supply and controller have enough slew rate to accommodate the control output changes needed for stability? Finally, what model are you using in your controller? A basic PID controller should be pretty good if the above items are okay, but if you still have nonsteady motion you might need to look into feedforward control, or maybe H$\infty$ control to get the stability you need.
I know this isn't a real answer, but hopefully it gives you some ideas.