1
$\begingroup$

I am posting this here because my background in estimation theory and optimization has been developed entirely through my experience in robotics.

TLDR: What makes it so that any time you put something in an exponential you can suddenly call it a probability distribution?

The Deets: I thought of this question while reading A General and Adaptive Robust Loss Function by Jonathan Barron. The paper describes a single parameter loss function that can subsume the most commonly used individual loss functions:

$$f(x, \alpha, c)$$

Where $x$ is parameter, $\alpha$ is the "single parameter", and $c$ is a distance scale that describes the "quadratic bowl" around the origin which all the loss functions share in common. When $\alpha = 2$ it is squared error loss, $\alpha =1$ it is essentially absolute error loss, and when $\alpha = 0$ it is cauchy loss.

My confusion starts when the paper goes on to say "With our loss function we construct a general probability distribution".

$$ p(x|\mu,\alpha, c) = \frac{1}{cZ(\alpha)}\exp(-f(x-\mu, \alpha, c)) $$

My question is, why does dividing by a normalization constant, and taking the exponential of the negative loss function suddenly make it a pdf? I understand the results, I have used these same ideas in my estimation theory courses, but I just don't get how we can make the jump from loss to pdf OR why it is so intuitive to do so.

Cheers, and happy holidays :)

$\endgroup$
2
  • $\begingroup$ The loss function was introduced by Jonathan Barron to solve an optimization problem. There is a model, and adapting the model to the real world produces a loss. The task for the solver is to reduce the costs to a minimum. A probability distribution helps to apply existing search techniques likes gradient descent, so that the minimum value can be found faster. $\endgroup$ – Manuel Rodriguez Dec 25 '20 at 18:14
  • 2
    $\begingroup$ You would probably have more luck with this question on stats.stackexchange.com or ai.stackexchange.com $\endgroup$ – 50k4 Dec 27 '20 at 19:59
3
$\begingroup$

I believe it's because you're essentially constructing an exponential distribution which has the form

expontial distribution pdf

Because your loss function will always be >= 0, you form a valid PDF (valid in that it integrates to 1, but your loss function might not make that practically true)

$\endgroup$
2
$\begingroup$

To complement what Octopuscabbage correctly reported, there exists a strong theoretical foundation for using normal probability distributions in many different contexts, which builds on the Central Limit Theorem (CLT) that explains how the "exponential" distribution can work well with problems involving other types of distributions.

As a result, when you don't know the probability distribution of a random variable, picking up the normal distribution turns out to be the most preferable choice.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.