Your code assumes that the CPU is running at a constant clock. However, in real life it will change, and sometimes even drastically! So it is required to insert a Delta Time (dt) factor to your KI and KD terms in the computations.
(KI_ROLL * (roll_old + roll) *dt) +
(KD_ROLL * (roll - roll_old) *dt)
(dt = time_now - time_previous)
By the way, your Integral term is a wrong one! As it stands, it's probably destabilizing the whole system. An integral controller should use a Integral_Roll_error variable, which is the addition of errors in time. For example:
Integral_Roll_error = Integral_Roll_error + roll - roll_old;
Running fast is important, but running a stable rate is required, especially if you want to be able to analyse your system, or do tests, compare different control logics, to add a Navigation algorithm, or payload control algorithm later on, etc.
Coming back to the question:
"I am having trouble interpreting the system response on varying the constants."
It would be beneficial to elaborate a little bit more on what sort of funny or difficulties you ran into. Adding some response charts from your tests would probably let the community help many in practical ways.
The second question asks about the rate limit of the motor control output command. This depends a lot on what sort of a performance you are expecting. The code is only computing the roll_command. How it is turned to motor commands is a relevant topic. In general, as long as the motor accelerations do not cause harm, you could make them as fast as you could.
The performance requirement of the controller (how fast it should stably bring the system to the desired state, how far it should/could overshoot, what sort of a damping it should have), derives the required motor_rate_limit. For example, lift off could be one of the performance requirements. From a standing position to leving the ground, if it takes 30 seconds, is it acceptable? or if the vehicle is tilted by 60 degrees in 2 seconds, (the lift will drop to half if not compensated), you need more Lift all of a sudden. So angular agility of the vehicle is indirectly affecting the lift_agility of the vehicle, and in turn the motor_control_rates.
3) At what thrust range or values, should the tuning be performed? Minimum, lift off, or mid-range or is it irrelevant?
It is relevant. At the limits of thrust, you are not able to accelerate the motors that are already at their maximum. (unless you never allow to go beyond a specific total_Thrust, which is a workaround, by giving up some performance.) So the gains of the controller must be tuned at the maximum, as well as some other points of the operational envelope (forward flight, hover). Reminder: Max thrust and similarly min thrust tests would not be easy to achieve with normal flights.