# SLAM : Why is marginalization the same as schur's complement?

Consider the system $$\tag 1 H\delta x=-g$$ where $H$ and $g$ are the Hessian and gradient of some cost function $f$ of the form $f(x)=e(x)^Te(x)$. The function $e(x)=z-\hat{z}(x)$ is an error function, $z$ is an observation (measurement) and $\hat{z}$ maps the estimated parameters to a measurement prediction.

This minimization is encountered in each iteration of many SLAM algorithms, e.g.one could think of $H$ as a bundle adjustment Hessian. Suppose $x=(x_1,x_2)^T$, and let $x_2$ be some variables that we seek to marginalize. Many authors claim that this marginalization is equivalent to solving a smaller liner system $M\delta x_1=-b$ where $M$ and $g$ are computed by applying Schur's complement to (1), i.e. if $$H= \begin{pmatrix} H_{11} & H_{12}\\ H_{21} & H_{22} \end{pmatrix}$$ then $$M=H_{11}-H_{12}H_{22}^{-1}H_{21}$$ and $$b=g_1-H_{12}H_{22}^{-1}g_2$$

I fail to understand why that is equivalent to marginalization... I understand the concept of marginalization for a Gaussian, and I know that schur's complement appears in the marginalization if we use the canonical representation (using an information matrix), but I don't see the link with the linear system.

Edit: I understand how Schur's complement appears in the process of marginalizing or conditioning $p(a,b)$ with $a,b$ Gaussian variables, as in the link supplied by Josh Vander Hook. I had come to the same conclusions, but using the canonical notation: If we express the Gaussian $p(a,b)$ in canonical form, then $p(a)$ is gaussian and its information matrix is the Schur complement of the information matrix of $p(a,b)$, etc. Now the problem is that I don't understand how Schur's complement appears in marginalization in bundle adjustment (for reference, in these recent papers: c-klam (page 3 if you want to look) and in this (part titled marginalization). In these papers, a single bundle adjustment (BA) iteration is performed in a manner similar to what I initially described in the question. I feel like there is a simple connection between marginalizing a Gaussian and the marginalization in BA that I am missing. For example, one could say that optimizing $f$ (one iteration) is equivalent to drawing a random variable following a denstiy $$e^{-\frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu)}$$ where $\Sigma$ is the inverse of the Hessian $H$ of $f$, and $\mu$ is the true value for $x$ (or an approximation of that value), and that marginalizing this density is equivalent to using Schur's compelement in the bundle? I am really confused...

• Are you asking why the schur complement falls out of the marginalization of a MV Gaussian? Or are you asking why the same form appears in the bundle-adjustment type optimization? – Josh Vander Hook Jan 16 '16 at 0:52
• Basically the problem is with why and how it appears in BA. I edited the question to clarify. – Ash Jan 16 '16 at 9:15
• Thanks for the clarification! I see now. I've updated my answer and hopefully others can come along and help out as well. At this point I think we've beat this to death. if you have further questions I suggest you submit a new question. – Josh Vander Hook Jan 16 '16 at 16:51

See the walk-through

The Schur complement helps with the closed form derivation but isn't necessary. It's just a nice convenient property of Gaussians and the covariance matrices.

In these papers, a single bundle adjustment (BA) iteration is performed in a manner similar to what I initially described in the question.

The reason the marginal / schur complement appears over and over is that all of the objective functions, noise models, and (in some sense) optimization methods are the same.

All the problems attempt to find a vector $x$ which maximizes the likelihood of the observations $z=h(x)+\eta$ where $\eta$ is some Gaussian noise. $h(x)$ is often not linear, but we linearize it so that $h(x)\approx h(x)+H_x(a-x)$ where (sorry for abusing notation), $H$ is the Jacobian.

I feel like there is a simple connection between marginalizing a Gaussian and the marginalization in BA that I am missing.

Everyone and their mother assumes that the noise in the observation $z$ is Gaussian. If you construct the Gaussian noise terms, you'll find something like $e^{ (z-h(\hat{x}))^T\Sigma^{-1}(z-h(\hat{x}) )}$ (modulo the denominator).

However, this isn't the only formulation. We can also minimize the error (difference in observations and state estimate). A careful look reveals that maximizing the likelihood of the observations is equivalent to finding a vector $x$ which minimizes the error $\eta=z-h(x)$.

Thus, minimizing the mean squared error $(\eta^T\eta)$ is the same as finding $x$ which maximizes the likelihood of the observations $z$. Thus, minimizing mean squared error and maximizing Gaussian observation likelihood results in the same objective function.*

If we substitute our linear approximation, we need to minimize

$$J=(z-h(x)+H_x(a-x))^T(z-h(x)+H_x(a-x))$$

If you multiply that out, and minimize by taking the derivative, you get the solutions you showed above. This is the same across any ML estimation of a state vector given Gaussian noise. The difference in various estimation methods is summarized as follows, as best as I know.

• The EKF does only one optimization step, and uses only the last measurement and the current state approximation
• Other methods, like IteratedEKF use many iterations of the solution, but only use the most recent measurement
• If you can afford to use more than one measurement, you've arrived at a larger state (one copy of the state vector for each time step, and a stacked vector of observations). Doing this optimization over this larger state is what BA does, but is also known as Iterative Weighted Least Squares. More generally, these are batch estimators.
• What's nice in the computer vision world, is that a huge state vector and observation vector have independent components (because a camera cannot see all parts of the world at all time). This structure is exploited to find what is classically known as Bundle Adjustment.
• If you wish to discard the Gaussian noise assumption and evaluate a larger space of possible soutions, you can use sampling based methods. However, almost all sampling based methods I'm aware of sample $x$ (the state), and run one of the above estimators for each $x$ using the observations, then find the posterior probability that the sample you drew was correct.

For example, one could say that optimizing ff (one iteration) is equivalent to drawing a random variable following a density ...

Here you're a bit off. That's not correct that we're drawing samples, but it is correct that by trying to minimize error, we're actually maximizing likelihood of the Gaussian. This is how the error minimization objective of BA, the Gaussian noise model (and resulting appearance of a marginalization i.e., schur complement), and general slam frameworks are all related. They're the same, just found by different methods across different disciplines. Isn't research great?

• *Kinda. In this derivation it does, but we do assume that the observation noise is Gaussian to arrive there. It's a difference in starting point, not path. The following image should help. It's an old post but just in case anyone comes here later.