With the problem of stabilising an inverted pendulum on a cart, it's clear that the cart needs to move toward the side the pendulum leans. But for a given angle $\theta$, how much should the cart move, and how fast? Is there a theory determining the distance and speed of the cart or is it just trial and error? I've seen quite a few videos of inverted pendulum, but it's not clear how the distance and speed are determined.
There are lots of ways to solve this problem, which falls into the category of Control Engineering. There are two standard approaches:
Classical Control: The control command has to be proportional to a linear combination of the error, the rate of change of the error, and the integral over time of the error, a.k.a. a PID controller. This approach requires you to tune three parameters controlling which dictate how much gain should be used in proportion to the three aforementioned measurements. This can be done manually by trial and error, but more sophisticated approaches exist such as Ziegler-Nichols tuning.
Optimal Control: construct a cost function which is constrained by the system's dynamics and obtain it's minimum. A common choice is a quadratic cost function in terms of the state vector and the control command, leading to a thing called an LQR controller. The minimum is obtained by solving the Riccati equations, for which there are standard numerical techniques.
In my personal experience, LQR gives better results, but requires a bit of computational power and mathematical understanding to accomplish. PID is, by far, easier to implement and doesn't require a whole lot of mathematical understanding to do, but may not perform as robustly as you wish.
In any case, you will need sensors to detect the pendulum's angle and either its rate of change in angle, or a data recorder to accumulate the error over time, or both.
Also, you may find that your sensors do not always give correct values, due to cheap technology or unexpected disturbances, in which case you also need to account for uncertainty in measurements. A Kalman filter would be needed in such a case, and a sophisticated mathematical understanding will of the dynamics be needed to develop one properly.
The theory that describes what you are looking for is call Control Theory. Search for the Nonlinear Systems textbook by Hassan Khalil for an excellent overview of the material--the inverted pendulum problem is addressed explicitly.
To theoretically stabilize the inverted pendulum on the cart, a model of the dynamics of the system are needed and can easily be found on the web. The next step is to find a state space representation of the dynamics. With the state space represented, there are two options--linearize the dynamics around the desired equilibrium (balanced pendulum bob angle of zero degrees from vertical) or utilize the nonlinear dynamics directly. Most projects that I have seen use linearization. At this point, a desired output must be chosen, and a controller accomplishing this objective must be found--this is the bread and butter of Control Theory, and it is non-trivial to do so. For linearized inverted pendulum systems, PID control is typically used.
So to answer your question: yes, there is theory; no, it isn't (shouldn't be) trial and error, and yes, it is hard to accomplish.