I'm not sure which way you're trying to model this; forward or inverse kinematics, but the term you should look for is "saturation". Saturation means basically that the system can't continue to respond despite a signal being present. For example, a 0-10V sensor being supplied with 11V is saturated because it will only show you 10V. Likewise, your actuators saturate at $\tau_{\mbox{max}}$ despite the fact that you are asking/signalling for more torque.
I don't know how you're simulating this system (Matlab/Simulink or ROS, etc.), but Simulink has a saturation block you can use on your actuators. If you're using other software you can implement saturation yourself:
torqueMax = <max torque>;
torque = <control loop output>;
if abs(torque) > torqueMax
torque = sign(torque) * torqueMax;
end
You can of course have two different saturation values, one for a maximum limit and one for minimum, you'd just use two if
statements to do the check.
Saturation adds nonlinear elements to your model, which may or may not be acceptable to you. If you don't want to deal with the nonlinearities this introduces, then you should lower your control gains such that your response slows down to the point that you never enter saturation.
I won't go into any more detail than to just say that "saturation is a thing" with a brief overview as your question seemed to be more in the scope of asking how to handle control/simulation outputs that exceed physical capability - I gave a code snippet above for that. If you want to read more on the subject there are a ton of articles on Google Scholar.