Your last objective
Name the loss you used most recently. Why that one, and who decided?
Inverse Problems and Learning
École Centrale Casablanca · 2026
What follows, what does not, and what assumptions select · C1
Hermann Agossou1,2
PhD candidate, UTER MID@S, ECC / UH2C
1École Centrale Casablanca · 2Université Hassan II de Casablanca
01
Part 01
The vocabulary
Two questions before we start
Name the loss you used most recently. Why that one, and who decided?
Had you chosen differently, what would have changed about the claim you made?
Most answers are about convenience or convention. By the end of the session that question has a precise answer, and it is not about convenience.
3 / 33
The vocabulary
They may be answering different questions
Targets a conditional mean under squared decision loss. A Gaussian likelihood is one stronger reading.
Targets a conditional median under absolute decision loss. Robustness is a separate empirical question.
Can encode a count-data likelihood when its support, rate, and dependence assumptions are actually stated.
The objective names what is optimized. It does not, by itself, identify a data-generating model or prove that the chosen decision target matches the application.
4 / 33
The vocabulary
Same expression, different variable held fixed
Write $p(D\mid\theta)$. Fix $\theta$ and vary $D$: this is a mass or density over possible data. Fix the observed $D$ and vary $\theta$: this is a likelihood. The formula does not change; its domain and normalization obligation do.
The symbolic version. Same right-hand side; only the variable being varied differs.
$\sigma$ is reserved for the unknown physical parameter in tracks B and D. Here the Gaussian noise scale is $\sigma_{\mathrm{n}}$, with $\varepsilon \sim \mathcal{N}(0,\sigma_{\mathrm{n}}^2)$; the Poisson rate is $r$; the Bernoulli probability is $\pi$. The Gaussian mean is written $f_\theta(x)$ because it is the model output.
5 / 33
The vocabulary
The notation everything else is written in
Discrete: p(x) is the probability of that exact outcome, in [0,1], summing to one. Continuous: the probability of any exact value is zero, so p(x) is a density; probability appears only on integrating. A density is not a probability and may exceed one.
discrete: $\sum_x p(x)=1$ · continuous: $\int p(x)\,dx=1$, while $p(x)$ itself may exceed 1
Both hold at once for a continuous variable, and neither contradicts the other.
A Gaussian with $\sigma_{\mathrm{n}} = 0.01$ has density about 40 at its peak. Nothing is wrong: it is large because it is concentrated on a narrow interval, and its integral is still one.
6 / 33
The vocabulary
The assumption that licenses a product
Given $\theta$, one observation provides no further information about another. They still share $\theta$; only the remaining randomness factorizes.
i.i.d. adds one shared law · unequal $s_i$ can be independent without being identical
The logarithm turns the licensed product into the sum optimized in code.
Nothing forces this factorization. It is a claim about the data, and part 4 deliberately removes it.
7 / 33
The vocabulary
A data density is not parameter probability
The likelihood is the sampling mass or density read as a function of the parameter with data fixed. It has no normalization obligation over $\theta$. A bare integral over $\theta$ depends on a chosen measure and parameterization; marginal likelihood instead integrates against a prior.
$L(\theta;D)$ need not integrate to one over $\theta$
See: [1] Casella & Berger (2002)
This is not pedantry. It is the difference between maximum likelihood and Bayesian inference, and it decides what your fitted number is allowed to be called.
8 / 33
02
Part 02
From model to loss
Predict, distribute, multiply, log, negate
Workflow 01
The model outputs $f_\theta(x)$.
Workflow 02
Say which distribution $f_\theta(x)$ parameterises.
Workflow 03
Build the joint likelihood under a stated dependence model.
Workflow 04
Turn licensed products into sums.
Workflow 05
Minimize the negative log-likelihood.
Maximising the likelihood and minimising this are the same problem: the logarithm is increasing, and the minus sign flips the direction.
10 / 33
From model to loss
Fully derived, then checked numerically
Sum scalar negative log-densities and compare with the vectorized NLL. A mismatch exposes a lost scale or normalization term.
11 / 33
From model to loss
The model is given, the algebra is yours
Given. The exponents simply select $\pi$ when $y=1$ and $1-\pi$ when $y=0$.
5 minutes. Artifact: one derivation per pair from the Bernoulli mass to binary cross-entropy, with the conditional-independence step boxed. Check it against a second pair before the reveal.
12 / 33
From model to loss
The multiclass case, derived in full
The one-hot product selects the true class. Compare the final sum with library cross-entropy on the same logits and targets.
13 / 33
From model to loss
Counts, and the term you are allowed to drop
The rate must stay positive. Compute the objective with and without log y!; the minimizer must be unchanged because that term is data-only.
14 / 33
From model to loss
Six entries, none optional
| Layer | Declared assumption | If changed |
|---|---|---|
| Support | $Y_i\in\mathbb{R}$ | Counts or classes require a different law |
| Measure | Lebesgue density | A discrete outcome uses probability mass |
| Conditional law | Gaussian around $f_\theta(x_i)$ | Tail and support commitments change |
| Parameterization | $f_\theta(x_i)$ is the conditional mean | A different target changes the interpretation |
| Dependence | Independent conditional on $X,\theta$ | Use a joint covariance, not a product |
| Scale | Known $s_i>0$, allowed to differ | Unknown or predicted scales add parameter terms |
15 / 33
From model to loss
Including where weight decay comes from
See: [2] Bishop (2006)
16 / 33
03
Part 03
Reading it backwards
One objective, several compatible readings
When the conditional second moment exists, expected squared loss is minimized by the conditional mean. No Gaussian assumption is needed.
Gaussian, conditionally independent, constant-variance errors make empirical squared error an NLL up to scale and constants.
An optimizer may use squared loss for tractability without claiming that it is a log-likelihood or the final decision cost.
The objective alone does not choose among these readings. Additional assumptions and the decision target do.
Compare: [3] Miller et al. (1993); [2] Bishop (2006)
18 / 33
Reading it backwards
One expression, at least two licensed readings
Likelihood reading: conditionally independent Gaussian errors with one fixed scale make the NLL an affine transform of J.
Masses 0.8, 0.1, 0.1 on values 0, 1, 9 give mean 1. Squared risk is 7.2 at decision 1 versus 8.2 at decision 0, with no Gaussian law.
19 / 33
Reading it backwards
Multiple readings, defended aloud
12 minutes. Artifact: a three-column audit table with at least two compatible interpretations, each interpretation's added assumption, and the evidence that could support or refute it.
20 / 33
Reading it backwards
Resume with one question
What did your objective fail to identify without extra assumptions?
21 / 33
Reading it backwards
MLE, MAP, and the regularizer that was a prior
MLE asks which $\theta$ makes the observed data most plausible. MAP asks the same question after multiplying in a prior belief about $\theta$ itself. Taking logs turns that product into a sum: your data term, plus a penalty.
$\tau$ = prior standard deviation on each parameter · small $\tau$ = a strong belief that $\theta$ is near zero
Under this chosen isotropic Gaussian prior, the negative log-prior is quadratic and $\lambda$ records its strength.
22 / 33
04
Part 04
When the simple picture breaks
What EM is actually completing
Sometimes each observation carries a hidden label you never see: which mixture component produced this point. That label is the latent variable. If someone handed you all the labels, fitting would be easy. That imagined dataset is the complete data. EM works by averaging over the labels you do not have.
$C_i$ = hidden component label of observation $i$ · $(y_i,C_i)$ = complete data · $\gamma_{ij}$ = responsibility
The observed likelihood is the complete one summed over labels. $\gamma$ is the responsibility, and is what the E step computes.
24 / 33
When the simple picture breaks
What the 1977 paper actually establishes
Workflow 01
Compute the responsibilities $\gamma$ under the current $\theta$, then form the expected complete-data log-likelihood.
Workflow 02
Exact EM maximises that expectation over $\theta$; generalized EM need only increase it.
Workflow 03
Each iteration does not decrease the observed-data likelihood.
An exact EM step, or a generalized EM step that increases Q, produces an observed-data likelihood at least as large as before. Parameter convergence and optimality require separate conditions.
$\ell(\theta^{(t+1)}) \ge \ell(\theta^{(t)})$ for every $t$: an inequality, not a limit
See: [4] Dempster et al. (1977)
25 / 33
When the simple picture breaks
Monotone is not the same as convergent
Left
By the 1977 paper.
Right
And what students assert anyway.
Convergence of likelihood values does not imply parameter convergence. Wu obtains stationary limit points under added compactness and continuity, and the unique MLE only under stronger conditions.
See: [5] Wu (1983)
26 / 33
When the simple picture breaks
The trace is evidence, not a global certificate
| initial means | initial log L | final means | final log L | all increments |
|---|---|---|---|---|
| (-4, 0) | -50.188 | (-4.000, 2.000) | -30.928 | non-negative |
| (0, 4) | -50.188 | (-2.000, 4.000) | -30.928 | non-negative |
| (-4, 4) | -46.837 | (-2.697, 2.697) | -34.976 | non-negative |
Recompute each initial likelihood, then check every `np.diff(trace)`. A decrease falsifies the implementation contract; three increasing traces do not certify parameter convergence or a global optimum.
27 / 33
When the simple picture breaks
Where the kernel enters, and what it encodes
A positive-semidefinite kernel $k(x,x')$ specifies the covariance between function values. Radial kernels may decay with distance; periodic, linear, and constant kernels need not. Adding independent observation noise contributes only a diagonal term.
$K_{ij}=k(x_i,x_j)$ · $K_y=K+\sigma_{\mathrm{n}}^2I$ · independence implies diagonal covariance, not necessarily identity
The latent function has been integrated out. Independence implies diagonal noise covariance; identity additionally requires one common unit variance.
28 / 33
When the simple picture breaks
Just the multivariate Gaussian density, logged
Step 1: after marginalizing the latent function, one joint Gaussian density replaces an independence product.
Step 2: data fit, complexity, constant. Because $\log|K_y|$ depends on $\theta$, it stays.
Compare with a library GP value on the same data. A constant offset exposes a mishandled $2\pi$ term.
29 / 33
When the simple picture breaks
Observations that are not independent
Written entirely in K_y, which was defined on the previous slide.
The first term rewards data fit and the second penalizes covariance volume. Their sum supports hyperparameter comparison within the chosen GP family, but does not certify that family or prevent overfitting when many hyperparameters are optimized.
See: [6] Rasmussen & Williams (2006)
30 / 33
When the simple picture breaks
Classify by what varies, normalizes, and is integrated
31 / 33
When the simple picture breaks
One chain, read in both directions
$\sigma$ stays the physical parameter; the noise scale is $\sigma_{\mathrm{n}}$; $\lambda$ is the regularization weight, which is where B1's Tikhonov term came from.
32 / 33
05
Part 05