Why Does an Exponential Make ANYTHING a Probability Distribution

Question

I am posting this here because my background in estimation theory and optimization has been developed entirely through my experience in robotics.

TLDR: What makes it so that any time you put something in an exponential you can suddenly call it a probability distribution?

The Deets: I thought of this question while reading A General and Adaptive Robust Loss Function by Jonathan Barron. The paper describes a single parameter loss function that can subsume the most commonly used individual loss functions:

$$f(x, \alpha, c)$$

Where $x$ is parameter, $\alpha$ is the "single parameter", and $c$ is a distance scale that describes the "quadratic bowl" around the origin which all the loss functions share in common. When $\alpha = 2$ it is squared error loss, $\alpha =1$ it is essentially absolute error loss, and when $\alpha = 0$ it is cauchy loss.

My confusion starts when the paper goes on to say "With our loss function we construct a general probability distribution".

$$ p(x|\mu,\alpha, c) = \frac{1}{cZ(\alpha)}\exp(-f(x-\mu, \alpha, c)) $$

My question is, why does dividing by a normalization constant, and taking the exponential of the negative loss function suddenly make it a pdf? I understand the results, I have used these same ideas in my estimation theory courses, but I just don't get how we can make the jump from loss to pdf OR why it is so intuitive to do so.

Cheers, and happy holidays :)

Answer 1

I believe it's because you're essentially constructing an exponential distribution which has the form

Because your loss function will always be >= 0, you form a valid PDF (valid in that it integrates to 1, but your loss function might not make that practically true)

Answer 2

To complement what Octopuscabbage correctly reported, there exists a strong theoretical foundation for using normal probability distributions in many different contexts, which builds on the Central Limit Theorem (CLT) that explains how the "exponential" distribution can work well with problems involving other types of distributions.

As a result, when you don't know the probability distribution of a random variable, picking up the normal distribution turns out to be the most preferable choice.

Answer 3

Just thought I'd point out that this doesn't always work. Suppose your loss function is $f(x)=\ln(|x|+1)$. Then $e^{-f(x)}=(|x|+1)^{-1}$ and the area under this curve is $\infty$ since

$$ \int_{\infty}^\infty (|x|+1)^{-1} \,\mathrm{d}x =2\int_0^\infty (x+1)^{-1} \,\mathrm{d}x =2\ln(x+1)|_0^\infty =\infty $$

Thus there is no way to normalize and get a pdf even though $e^{-f(x)}>0$ everwhere.

With respect to the $f(x,\alpha,c)$ described in your linked paper, the paper itself tells us that for $\alpha<0$ that the integral of $e^{-f(x)}$ does not converge (stated below equation 17) i.e. normalization is impossible.

On the other hand, for some intuition, the construction described in the paper is ubiquitous in statistical physics and is called the Gibbs measure/distribution associated to an energy function (loss function). This provides us some expectation that this sort of thing should frequently work and is worth a try.