← ML Foundations

Learn / ML Foundations

ML Foundations: Gradient Descent and Optimization

Foundational quiz on gradient descent updates, learning rates, stochastic variants, loss functions, and regularization.

Learning path

0%

0 of 4 sections marked complete · about 28 minutes

Learning objectives

What you will be able to explain

  • What is the standard gradient descent update rule?
  • Take one gradient descent step
  • What is a common effect of a learning rate that is too large?
  • What is a common effect of a learning rate that is too small?
  • Update a two-parameter model
  • Optimization facts: true or false

Section 01

What is the standard gradient descent update rule? to What is a common effect of a learning rate that is too large?

01

What is the standard gradient descent update rule?

Gradient descent subtracts a scaled gradient. The gradient points uphill, so subtracting it moves parameters toward lower loss.

Guided checkpoint

Choose the correct parameter update.

02

Take one gradient descent step

Apply wnew=w−ηg=5−0.1⋅8=4.2w_{new} = w - \eta g = 5 - 0.1 \cdot 8 = 4.2.

Guided checkpoint

Suppose the current parameter is w=5w = 5, the gradient is 88, and the learning rate is 0.10.1. Compute the updated parameter.

03

What is a common effect of a learning rate that is too large?

A large learning rate can make updates bounce around the minimum or even diverge entirely because the steps are too aggressive.

Guided checkpoint

Choose the best answer.

Section 02

What is a common effect of a learning rate that is too small? to Optimization facts: true or false

01

What is a common effect of a learning rate that is too small?

Tiny learning rates make each parameter update very small, so optimization can take many steps to make progress.

Guided checkpoint

Choose the best answer.

02

Update a two-parameter model

The gradient is (2w1,4w2)(2w_1, 4w_2), so at (1,−1)(1, -1) it is (2,−4)(2, -4). One step gives (1,−1)−0.25(2,−4)=(0.5,0)(1, -1) - 0.25(2, -4) = (0.5, 0).

Guided checkpoint

Let f(w1,w2)=w12+2w22f(w_1, w_2) = w_1^2 + 2w_2^2. Start at (1,−1)(1, -1) and use one gradient descent step with learning rate 0.250.25.

03

Optimization facts: true or false

These statements summarize the tradeoffs among common gradient-based optimizers. Nonconvex objectives are especially tricky because local minima and saddle points may appear.

Guided checkpoint

Mark each statement as true or false.

Section 03

What is mean squared error (MSE)? to One gradient step for a regression example

01

What is mean squared error (MSE)?

MSE penalizes larger errors more strongly because it squares residuals. It is one of the standard losses for regression.

Guided checkpoint

Choose the correct description.

02

What is a common effect of L2L_2 regularization?

Adding an L2L_2 penalty discourages very large weights. In practice this often improves generalization by controlling model complexity.

Guided checkpoint

Choose the best interpretation.

03

One gradient step for a regression example

The gradient is dLdw=2(wx−y)x\frac{dL}{dw} = 2(wx - y)x. At w=1w = 1, this is 2(2−6)⋅2=−162(2 - 6) \cdot 2 = -16. The update is 1−0.05(−16)=1.81 - 0.05(-16) = 1.8.

Guided checkpoint

For the loss L(w)=(wx−y)2L(w) = (wx - y)^2 with a single example (x,y)=(2,6)(x, y) = (2, 6), start from w=1w = 1 and use learning rate 0.050.05. Compute the updated parameter after one gradient descent step.

Section 04

Which is a reasonable optimization stopping signal?

01

Which is a reasonable optimization stopping signal?

In practice we often stop when progress has effectively stalled, either because gradients are tiny or because validation performance no longer improves.

Guided checkpoint

Choose the best practical criterion.

Knowledge check

Turn understanding into recall.

The quiz now follows the same concepts in scored form. You can return to this lesson from the quiz whenever a gap appears.

Start the quiz →Back to course