Most often tuning the Kalman filter noise matrices is done by trial and error or domain knowledge. Are there more principled ways for tuning all the Kalman filter parameters?
You can get experimental data, and perform some statistical analysis to determine the process noise (noise between time steps), and sensor noise (compared to a ground truth).
To get the ground truth for sensor noise, you either need a more accurate sensor, or else experimentally test while keeping the state of interest at a known (usually fixed) value.
If you do not have experimental data, you can use the datasheets or specifications for each sensor to determine its noise. The process noise would be more difficult in this instance, but you may be able to guess at its value by assuming a certain amount of noise due to, for example, an actuator, if the actuator is the primary source of process noise.
Another way is to consider the maximum settling time allowed for estimation. This will determine the ratio between the process noise and sensor noise. This allows you to have the maximum filtering effect while meeting your settling time requirements.
Note that this does not provide a good way to tune the ratio between different sensors, or between different states, so is not perfect.
Basically, if you have experimental data, you can obtain optimal performance. Otherwise, you can only calculate process noise based by assuming the likely noise of the dominant contributor to process noise, or by determining what minimum performance is required (which is not optimal, but at least gets what is needed).
In the field of machine learning, we look at a Kalman filter as an inference algorithm on a latent variable model. The measurements are visible, but the true state is hidden. You now want to infer the true states.
As usual, this method relies on a set of parameters and -- in principle -- the way to obtain the best point-estimate for the parameters is to look at the likelihood for the data (given the parameters) and optimising your parameters to obtain the maximum value. This gives you the maximum likelihood estimate.
Unfortunately, in the case of the Kalman filter, this approach is not so easy and intractable to do in closed form. This is because we do not observe the hidden states. Therefore, one has to apply some tricks. One of them is to use the Expecation Maximisation (EM) Algorithm, which is an iterative approach: In the first step, you calculate the most likely value for your hidden states (the expected value) under your current parameters. In the second step, you keep the hidden states fixed and optimise the parameters to obtain the solution with highest likelihood. These steps are repeated until convergence.
You can read up on the general concept of EM in a whole bunch of text books on machine learning (only one example: "Bayesian Reasoning and Machine Learning" by David Barber). The process of inferring the parameters for linear dynamical systems (which gives you the Kalman filter type of model) is well described in a tech report by Zoubin Ghahramani and Geoffrey Hinton.
You might want to read "A Tool for Kalman Filter Tuning" by Bernt M. Åkesson, et. al. It describes the autocovariance least-squares (ALS) method.