What you're asking isn't going to be very easy with a standard RC servo.
What you're asking for is a back-drivable servo. I.E. one which you can freely rotate by applying an external torque. It is certainly possible to create these, and they are used on many robots, but most RC servos require considerable torque to back drive them. I would call them semi-backdrivable.
What prevents you back driving them? Two things:
Friction: Firstly, friction in the gear chain and motor. RC servos are always geared down; the output spins slower than the motor. Of course this means that when you try to back drive it, you have to spin the motor very quickly. Any friction or cogging in the motor will be felt dozens of time over at the output.
Current: something surprising about electric motors is that, if you short the terminals, they become harder to turn. The rotation of the motor generates an electric current which works against your rotation. The electronics inside the servo might well allow enough current to flow, even when off, that it noticeably prevents backdrivability. In the past I have noticed that some large stepper motors, even when ungeared, are almost impossible to back drive when plugged in to unpowered equipment. But when you unplug them, they can be spun freely.
So, one way you can improve the back-drivability is to prevent current flowing. The obvious way to do this is to fully disconnect the motor from its electronics. But this is hard to do, even using FETs, because the diodes inside the FETs might well allow current to flow. However, you could use a relay, which really would disconnect the motor. You'd only have to use it on one of the motor's terminals.
'Proper' servo controllers (like the ESCON from Maxon) actually contain a current controller which can actively prevent current flowing by applying the correct voltage at the motor terminals. Replacing the electronics in the servo with something capable of current control might really help.
What you should do: Open up the servo, unsolder the motor, and re-assemble it. How easy is it to back-drive now? If it's easy, then it may be possible to do what you want. If it's still too hard for your application, then you're going to have to choose a different servo, or make your own. Make one with a single gear stage so that the gear friction is reduced as much as possible, and use drive electronics with a current control loop to guarantee zero current flow.