In an ideal scenario, yes, that should be the case. When gravity compensation is implemented on robots, all joints apply a torque to balance out the torque applied by the force of gravity. They should ideally turn into floating robots objects.
However, it is not always the case due to inaccurate modeling and gravity compensation implementation. Moreover, to swing a robot arm, you'll also need to overcome friction, stiction and other forces that oppose motion at the joint level.
If you have a Zero-Friction Zero-Gravity controller (you feed gravity compensation torques as well as friction torques in a tight closed loop), then the robot should be simply floating in space, massless. In such a case, you can definitely move a robot arm around with your hand, regardless of the mass. It is because all the forces that the robot needs to overcome and break into motion have already been overcome and the slightest of external efforts shall be extra input and sufficient to move the robot.