Quizmaster

Learning objectives

What you will be able to explain

Regularization changes the learning problem
Parameter norm penalties
Gradient of a weight-decay term
Penalty versus constraint
Regularization in under-constrained problems
Dataset augmentation as prior knowledge

Section 01

Regularization changes the learning problem to Gradient of a weight-decay term

Regularization changes the learning problem

Regularization is broader than explicit penalties and includes data, architecture, optimization, noise, ensembles, and invariance-based interventions.

Guided checkpoint

Which definition best matches the chapter?

Source: Chapter 7, p. 228

Parameter norm penalties

L^2

and

L^1

impose different geometry. The effective shrinkage also depends on the training objective's local curvature.

Guided checkpoint

Match each penalty or effect to its description.

Source: Chapter 7, section 7.1, pp. 230-237

Gradient of a weight-decay term

The gradient of

(\alpha/2)||w||^2

\alpha w

, giving

0.1(4)=0.4

Guided checkpoint

A regularized objective adds $(\alpha/2)||w||_2^2$ with $\alpha=0.1$ . For a coordinate $w_i=4$ , what additive gradient contribution comes from the penalty?

Source: Chapter 7, section 7.1.1, pp. 231-234

Section 02

Penalty versus constraint to Dataset augmentation as prior knowledge

Penalty versus constraint

A Lagrange multiplier connects a norm ball constraint to a penalized objective, though the mapping between penalty strength and radius is data-dependent.

Guided checkpoint

Under suitable regularity conditions, what constrained problem corresponds to an explicit norm penalty?

Source: Chapter 7, section 7.2, pp. 237-239

Regularization in under-constrained problems

Deep models are often underdetermined. Generalization depends on which solution is selected, not only whether training data are fit.

Guided checkpoint

Judge each statement.

Source: Chapter 7, section 7.3, pp. 239-240

Dataset augmentation as prior knowledge

Augmentation encodes known invariances. A transformation that changes the semantic target injects label noise rather than helpful prior knowledge.

Guided checkpoint

Which statements are correct?

Source: Chapter 7, section 7.4, pp. 240-242

Section 03

Noise as regularization to Early stopping protocol

Noise as regularization

Different noise mechanisms impose different robustness assumptions; adversarial training explicitly uses locally damaging perturbations.

Guided checkpoint

Match each injection site to the intended effect or interpretation.

Source: Chapter 7, sections 7.5 and 7.13, pp. 242-243 and 268-270

Semi-supervised and multi-task regularization

All these methods use additional structure beyond one labeled task to restrict or inform the learned solution.

Guided checkpoint

Match each setup to the source of extra constraint.

Source: Chapter 7, sections 7.6-7.7 and 7.9, pp. 243-246 and 253-254

Early stopping protocol

Early stopping treats training time as a capacity-like hyperparameter. The best validation checkpoint is selected, not merely the last iterate.

Guided checkpoint

Which checkpoint should be retained when validation loss first improves and later deteriorates while training loss continues falling?

Source: Chapter 7, section 7.8, pp. 246-253

Section 04

Why early stopping regularizes to Variance of an averaged ensemble

Why early stopping regularizes

Early stopping is computationally attractive but still needs validation monitoring and checkpointing or retraining strategy.

Guided checkpoint

Evaluate each claim.

Source: Chapter 7, section 7.8, pp. 246-253

Sparse parameters versus sparse representations

A representation penalty acts on unit activations, whereas parameter sparsity acts on the model's weights. Their computational and statistical effects differ.

Guided checkpoint

Which distinction is correct?

Source: Chapter 7, section 7.10, pp. 254-256

Variance of an averaged ensemble

For correlated equal-variance errors,

Var(average)=\sigma^2(1/k+(k-1)\rho/k)=9(0.25+3(0.25)/4)=3.375

Guided checkpoint

Each of $k=4$ predictors has error variance 9 and pairwise error correlation 0.25. Assuming equal variance, compute the variance of their average.

Source: Chapter 7, section 7.11, pp. 256-258

Section 05

Why ensembles help to Inverted-dropout expectation

Why ensembles help

The gain comes from error cancellation across diverse members; perfectly correlated errors do not average away.

Guided checkpoint

Select all correct statements.

Source: Chapter 7, section 7.11, pp. 256-258

Dropout as model averaging

Dropout shares parameters across an exponential family of subnetworks. Weight scaling provides an efficient approximation to their ensemble prediction.

Guided checkpoint

Which interpretation best matches the chapter?

Source: Chapter 7, section 7.12, pp. 258-268

Inverted-dropout expectation

The value is

6/0.75=8

when kept and 0 otherwise, so the expectation is

0.75(8)=6

Guided checkpoint

An activation is $h=6$ , kept with probability $0.75$ , and divided by the keep probability when retained. What is its expected post-mask value during training?

Source: Chapter 7, section 7.12, pp. 258-268

Section 06

Tangent-based regularization

The desired invariance is selective: predictions should remain stable along class-preserving manifold directions, not necessarily along arbitrary perturbations.

Guided checkpoint

Judge each statement.

Source: Chapter 7, section 7.14, pp. 270-273

Knowledge check

Turn understanding into recall.

The quiz now follows the same concepts in scored form. You can return to this lesson from the quiz whenever a gap appears.

Start the quiz →Back to course