I've hacked a rc helicopter, and I am able to control it by running a program on my computer. I am interested in writing algorithms that will stabilize the helicopter. For instance, the helicopter is hovering, and then if it is shoved off balance it can return to its previous position in a stable state. Any help on an algorithm would be awesome.
Stabilization of a helicopter and a quadrotor are similar tasks - have a reference signal, compare that to feedback, then act on the difference.
A quad rotor has four motors, and the helicopter arguably does as well: main rotor, tail rotor, swash plate fore/aft servo, swash plate port/stbd servo.
I would bet you can find a helicopter community that could point you to stabilization algorithms specific to helicopters, but it might be easier to build on the (freely available and widely published) resources available for quadrotors.
For instance, you'll need to communicate with a transmitter. Clearly you have that already. You'll need some form of an IMU; a microcontroller to communicate with that, the receiver, and the motors; electronic speed controllers if you find overriding the existing circuit is too difficult; etc.
Once you've got all that, you'll need the code to interface with all of the above.
The only difference really comes down to your model of how a particular control axis affects the motion of the vehicle. You could do modeling in state space, but I think most quadrotors just use PID control on each axis and tune the gains to achieve the desired response.
You can do the same; there will only be a small change. A quadrotor use PID controllers on each axis and then sums those outputs to give values for each motor, because the motions are coupled. Your roll, pitch, yaw, and thrust are all separate, so each controller runs its own motor/servo - no summing.
So, tl;dr - use widely available quadrotor framework, assign each axis to one motor, then do some PID tuning.