When setting up an optimal control problem using a linear quadratic regulator, one often chooses semi-definite matrices Q and R such that the objective function J has the form

$$J=\int_{t_0}^{t_f}(x^TQx+u^TRu)dt$$

I can see how choosing Q and R as diagonal matrices is advantageous because they make the drive the state x and control values to be as small as possible (close to zero). I also understand that a positive definite or semi definite matrix can also (at least in principle) be used because it would make the objective function essentially still a convex parabola whose minimum is still a single point, which also helps to drive the state and controls to be as small as possible.

In practice, I can’t think of a situation where a non-diagonal semi definite matrix would be better than a diagonal one for an LQR control problem. What kinds of problems/ situations make it more advantageous to choose off-diagonal entries of Q and R to be non-zero? I want to be able to identify situations that may come up in my work where this might be necessary, so I’m more interested in guiding principles rather than individual situations or exceptions.

In general it not meaning full to say that a diagonal $$Q$$ is advantageous, since one can always perform a similarity transformation to the state space model ($$x_\text{new}=T\,x$$) which would make the effective $$Q$$ ($$x^\top Q\,x = x_\text{new}^\top T^{-\top} Q\,T^{-1} x_\text{new}$$, so the effective $$Q$$ would be $$T^{-\top} Q\,T^{-1}$$) for the new state coordinates non-diagonal. In other words given any non-diagonal semi-positive definite $$Q$$ there is always a similarity transformation which would make the effective $$Q$$ for the new state coordinates diagonal.
As for $$R$$ it would depend on what you would like to minimize. For example if $$u$$ represents two parallel actuators and you want to minimize their combined force. This can be written as
$$(w_1\,u_1 + w_2\,u_2)^2 = u^\top \begin{bmatrix} w_1^2 & w_1\,w_2 \\ w_1\,w_2 & w_2^2 \end{bmatrix} u,$$
with $$w_1,w_2$$ some weights. Though normally $$R$$ should be positive definite (not just semi-positive definite), so one would have to add some additional term to ensure this, but this should illustrate how one could get to an non-diagonal $$R$$.