LQR design with low effort

I am trying to implement a controller for an inverted pendulum using LQR (with MATLAB command lqr(A,B,Q,R)). The problem is that the motors are relatively weak, so I tried to increase R, but simulations show that the effort is still very high. How can I reduce the effort?

With LQR you can either decrease $Q$ or increase $R$. If that's not working you have to consider the possibility that your motor just will not work.
It also might be worth looking at different controller design techniques. If I remember correctly, LQR minimizes the total energy, which usually means a large initial correction followed by a slop back to zero for the control signal (as opposed to lots of oscillation). There are other techniques that can focus on the maximum peak instead of total energy (e.g. generic optimal control or $\mathcal{H}_2$ robust-control techniques, which Matlab has commands for) but they tend to be difficult to understand and use without a lot of study.