Quizmaster

Learning objectives

What you will be able to explain

Why a linear model cannot learn XOR
Roles in a feedforward network
Sigmoid output and cross-entropy
Choose the output unit
Softmax invariance
Hidden-unit behavior

Section 01

Why a linear model cannot learn XOR to Sigmoid output and cross-entropy

Why a linear model cannot learn XOR

A hidden layer can learn a transformed representation in which the output becomes linearly separable.

Guided checkpoint

What geometric fact prevents a single affine decision boundary from representing XOR?

Source: Chapter 6, section 6.1, pp. 171-177

Roles in a feedforward network

A feedforward network composes parameterized functions without feedback connections; hidden units mediate learned representations.

Guided checkpoint

Match each object to its role.

Source: Chapter 6, pp. 168-171

Sigmoid output and cross-entropy

Matching the output distribution and log-likelihood yields well-behaved gradients even when the sigmoid saturates in the wrong direction.

Guided checkpoint

For binary classification with a sigmoid output, why is cross-entropy usually preferable to squared error?

Source: Chapter 6, section 6.2.1, pp. 178-186

Section 02

Choose the output unit to Hidden-unit behavior

Choose the output unit

The output layer should parameterize a valid conditional distribution; its activation and cost function are therefore chosen together.

Guided checkpoint

Match each predictive distribution to the natural output parameterization.

Source: Chapter 6, section 6.2.2, pp. 180-191

Softmax invariance

The ratio is

e^{z_2}/e^{z_1}=e^{z_2-z_1}

, so adding a common constant cancels and the ratio is unchanged.

Guided checkpoint

For logits $[1,2]$ , add 100 to both logits. By what multiplicative factor does the probability ratio $p_2/p_1$ change?

Source: Chapter 6, section 6.2.2.3, pp. 186-189

Hidden-unit behavior

Activation choice shapes gradient flow and representational behavior. The book recommends ReLU-like defaults for many feedforward hidden layers.

Guided checkpoint

Match each activation to its defining form or training characteristic.

Source: Chapter 6, section 6.3, pp. 191-197

Section 03

Rectifier variants to Why use a computational graph?

Rectifier variants

Nondifferentiability at isolated points is handled by choosing a subgradient or implementation convention; it is not normally an obstacle to gradient-based training.

Guided checkpoint

Judge each statement.

Source: Chapter 6, section 6.3.1, pp. 193-195

Depth, width, and universal approximation

Existence of a representation is distinct from optimization, statistical efficiency, and compactness. Depth can offer compositional efficiency.

Guided checkpoint

Select all claims supported by the architecture discussion.

Source: Chapter 6, section 6.4, pp. 197-204

Why use a computational graph?

The graph exposes dependencies, enabling systematic forward evaluation and efficient chain-rule accumulation.

Guided checkpoint

What is its central role in differentiation?

Source: Chapter 6, section 6.5, pp. 204-224

Section 04

Differentiation strategies to Back-propagation implementation facts

Differentiation strategies

Back-propagation is a specialized reverse-mode automatic-differentiation procedure on a computational graph.

Guided checkpoint

Match each strategy to its key tradeoff.

Source: Chapter 6, sections 6.5.2-6.5.5, pp. 207-221

Trace a scalar chain rule

dy/dv=3

dv/du=2u

, and

du/dx=2

. At

x=2

u=4

, so the product is

3(8)(2)=48

Guided checkpoint

Let $u=2x$ , $v=u^2$ , and $y=3v$ . Compute $dy/dx$ at $x=2$ .

Source: Chapter 6, section 6.5.1, pp. 204-207

Back-propagation implementation facts

Differentiation and optimization are distinct. Memory can be traded for recomputation, and fan-out requires summing gradient contributions.

Guided checkpoint

Evaluate each statement.

Source: Chapter 6, section 6.5, pp. 204-224

Section 05

Beyond ordinary real arithmetic to Historical perspective

Beyond ordinary real arithmetic

Autodiff systems compose local vector-Jacobian products. This abstraction extends reverse-mode differentiation to many tensor operations.

Guided checkpoint

What permits back-propagation through a new primitive operation?

Source: Chapter 6, section 6.5.6, pp. 221-224

Historical perspective

The historical notes separate gradient computation from optimization and place modern practice in a longer development of neural-network and autodiff ideas.

Guided checkpoint

Mark each statement as true or false.

Source: Chapter 6, section 6.6, pp. 224-227

Knowledge check

Turn understanding into recall.

The quiz now follows the same concepts in scored form. You can return to this lesson from the quiz whenever a gap appears.

Start the quiz →Back to course