Quizmaster

Learning objectives

What you will be able to explain

What does least squares minimize?
Fit a line through two points
Predict with the fitted line
Compute a residual
Compute the gradient of a small squared-error objective
Which formula gives the normal-equation solution?

Section 01

What does least squares minimize? to Predict with the fitted line

What does least squares minimize?

Least squares minimizes squared errors between predictions and targets. This produces a convex objective and leads to the normal equations in the linear case.

Guided checkpoint

Choose the standard objective.

Fit a line through two points

The point with

x = 0

gives

w_0 = 1

. Then

1 + 2w_1 = 5

implies

w_1 = 2

. With two points and two parameters, the line can match them exactly unless they are degenerate.

Guided checkpoint

Consider the model $\hat{y} = w_0 + w_1 x$ . Find parameters that exactly fit the two points $(0, 1)$ and $(2, 5)$ .

Predict with the fitted line

Substituting

x = 3

gives

\hat{y} = 1 + 2 \cdot 3 = 7

Guided checkpoint

Using $\hat{y} = 1 + 2x$ , compute the prediction for $x = 3$ .

Section 02

Compute a residual to Which formula gives the normal-equation solution?

Compute a residual

The residual is the observed value minus the predicted value, so

r = 8 - 7 = 1

. Squaring residuals gives the least-squares objective.

Guided checkpoint

Define the residual as $r = y - \hat{y}$ . If the true target is $y = 8$ and the prediction is $\hat{y} = 7$ , what is the residual?

Compute the gradient of a small squared-error objective

(0, 0)

the predictions are

(0, 0)

, so the errors

(\hat{y} - y)

are

(-1, -3)

. Summing derivatives gives

\partial L/\partial w_0 = -1 + (-3) = -4

and

\partial L/\partial w_1 = (-1)\cdot 0 + (-3)\cdot 1 = -3

Guided checkpoint

Use the model $\hat{y} = w_0 + w_1 x$ and the loss $L = \frac{1}{2}\sum_i (\hat{y}_i - y_i)^2$ on data $(x, y) = (0, 1)$ and $(1, 3)$ . Compute the gradient at $(w_0, w_1) = (0, 0)$ .

Which formula gives the normal-equation solution?

Setting the gradient of the least-squares objective to zero gives the normal equations

X^\top X w = X^\top y

, whose solution is

(X^\top X)^{-1}X^\top y

when invertible.

Guided checkpoint

Choose the closed-form least-squares solution when $X^\top X$ is invertible.

Section 03

Compute entries of $X^\top X$ to Linear regression facts: true or false

Compute entries of $X^\top X$

The first column is

(1,1,1)^\top

and the second is

(0,1,2)^\top

. Their inner products produce

X^\top X = \begin{bmatrix} 3 & 3 \\ 3 & 5 \end{bmatrix}

Guided checkpoint

Let the design matrix be $X = \begin{bmatrix} 1 & 0 \\ 1 & 1 \\ 1 & 2 \end{bmatrix}$ . Compute the distinct entries of $X^\top X$ .

What is the geometric meaning of least squares?

The least-squares prediction is the orthogonal projection of

y

onto the span of the columns of

X

. The residual is orthogonal to that column space at the optimum.

Guided checkpoint

Choose the best interpretation.

Linear regression facts: true or false

Least squares remains well defined even when exact fitting is impossible. Rank deficiency breaks uniqueness, ridge stabilizes the system, and the linear-regression objective is convex in the parameters.

Guided checkpoint

Mark each statement as true or false.

Section 04

Differentiate an $L_2$ penalty to What issue appears when the columns of $X$ are linearly dependent?

Differentiate an $L_2$ penalty

Because

R(w) = \lambda \sum_i w_i^2

, its gradient is

2\lambda w

. Here that gives

2 \cdot 0.1 \cdot (2, -1) = (0.4, -0.2)

Guided checkpoint

Let the regularization term be $R(w) = \lambda \|w\|_2^2$ with $w = (2, -1)$ and $\lambda = 0.1$ . Compute $\nabla R(w)$ .

What issue appears when the columns of $X$ are linearly dependent?

Linear dependence means some columns do not add new information, so different parameter vectors can produce the same predictions. This is why rank and regularization matter.

Guided checkpoint

Choose the best answer.

Knowledge check

Turn understanding into recall.

The quiz now follows the same concepts in scored form. You can return to this lesson from the quiz whenever a gap appears.

Start the quiz →Back to course

What you will be able to explain

What does least squares minimize? to Predict with the fitted line

What does least squares minimize?

Fit a line through two points

Predict with the fitted line

Compute a residual to Which formula gives the normal-equation solution?

Compute a residual

Compute the gradient of a small squared-error objective

Which formula gives the normal-equation solution?

Compute entries of X⊤XX^\top XX⊤X to Linear regression facts: true or false

Compute entries of X⊤XX^\top XX⊤X

What is the geometric meaning of least squares?

Linear regression facts: true or false

Differentiate an L2L_2L2​ penalty to What issue appears when the columns of XXX are linearly dependent?

Differentiate an L2L_2L2​ penalty

What issue appears when the columns of XXX are linearly dependent?

Turn understanding into recall.

Compute entries of $X^\top X$ to Linear regression facts: true or false

Compute entries of $X^\top X$

Differentiate an $L_2$ penalty to What issue appears when the columns of $X$ are linearly dependent?

Differentiate an $L_2$ penalty

What issue appears when the columns of $X$ are linearly dependent?