Problem 30 - Calculus Early Transcendentals: Pearson New International Edition

Question

Least Squares Given \(n\) points \(\left(x_{1}, y_{1}\right),\left(x_{2}, y_{2}\right), \ldots\), \(\left(x_{n}, y_{n}\right)\) in the \(x y\)-plane, we wish to find the line \(y=m x+b\) such that the sum of the squares of the vertical distances from the points to the line is a minimum; that is, we wish to minimize $$ f(m, b)=\sum_{i=1}^{n}\left(y_{i}-m x_{i}-b\right)^{2} $$

Step-by-Step Solution

Answer

Use calculus to derive equations, solve using normal equations, and find the line's slope \(m\) and y-intercept \(b\).

Step 1: Define the Objective Function

The goal is to find the parameters \(m\) (slope) and \(b\) (y-intercept) for the line \(y = mx + b\) such that the function \(f(m, b) = \sum_{i=1}^{n} (y_i - mx_i - b)^2\) is minimized. This function represents the sum of the squared vertical distances from each point \((x_i, y_i)\) to the line.

Step 2: Use Calculus to Minimize the Function

To find the minimum, we take the partial derivatives of \(f(m, b)\) with respect to \(m\) and \(b\) and set them equal to zero. This gives us two equations. 1. \( \frac{\partial f}{\partial m} = -2 \sum (y_i - mx_i - b)x_i = 0 \)2. \( \frac{\partial f}{\partial b} = -2 \sum (y_i - mx_i - b) = 0 \)

Step 3: Solve the System of Equations

Solve the following system of linear equations obtained from the conditions in Step 2:1. \( \sum x_i y_i = m \sum x_i^2 + b \sum x_i \)2. \( \sum y_i = m \sum x_i + nb \)These can be solved simultaneously using substitution or matrix methods such as the normal equations in linear algebra.

Step 4: Use Normal Equations to Find Values of m and b

Rewrite the system of equations in matrix form:\[\begin{pmatrix} \sum x_i^2 & \sum x_i \ \sum x_i & n \end{pmatrix} \begin{pmatrix} m \ b \end{pmatrix} = \begin{pmatrix} \sum x_i y_i \ \sum y_i \end{pmatrix}\]Solve for \(m\) and \(b\) by inverting the matrix or using techniques to find the solution.

Step 5: Calculate m and b

After finding \(m\) and \(b\), substitute the values back into the equation of the line \(y = mx + b\) to obtain the best fit line.

Step 6: Verify the Solution

Check if the line obtained indeed minimizes the sum of the squared distances by plugging a few points back into the line equation and confirming that the squared deviations are minimized.

Key Concepts

Objective Function

In the context of least squares regression, the objective function is the mathematical expression we aim to minimize to find the best-fitting line for a set of data points. The formula used is \( f(m, b) = \sum_{i=1}^{n} (y_i - mx_i - b)^2 \). Here,

• \(m\) represents the slope of the line,
• \(b\) is the y-intercept,
• \((x_i, y_i)\) are the given data points,
• The expression \((y_i - mx_i - b)^2 \) represents the square of the distance from each point to the line.

By minimizing this function, we ensure that the line will have the smallest possible sum of squared vertical distances from the data points. This objective is fundamental in creating a line that closely fits the data, reducing the total error between the observed and predicted values.

Partial Derivatives

Once the objective function is defined, the next step is to find its minimum by using calculus. This involves taking the partial derivatives of the function with respect to its parameters, \(m\) and \(b\). A partial derivative measures how the function changes as one of the variables changes, while keeping others constant. For our function,

• The partial derivative with respect to \(m\) is \( \frac{\partial f}{\partial m} = -2 \sum (y_i - mx_i - b)x_i = 0 \).
• The partial derivative with respect to \(b\) is \( \frac{\partial f}{\partial b} = -2 \sum (y_i - mx_i - b) = 0 \).

By setting these derivatives equal to zero, we find the critical points where the function's slope is zero, indicating potential minima or maxima. In the case of least squares, these points represent the optimal values for \(m\) and \(b\) that minimize the function.

System of Linear Equations

After taking partial derivatives and equating them to zero, we derive a system of linear equations. These equations are:

• \( \sum x_i y_i = m \sum x_i^2 + b \sum x_i \)
• \( \sum y_i = m \sum x_i + nb \)

This system arises from substituting the conditions from the partial derivatives. Solving this system allows us to find the exact values of \(m\) and \(b\).
The equations can be solved using algebraic methods such as substitution. Alternatively, they can be resolved using a more streamlined mathematical approach known as the matrix method. These solutions give us the critical values for the least squares fitting line.

Matrix Method

The matrix method is a powerful tool for solving systems of equations, especially in linear regression problems like least squares. We express the system of linear equations in a compact form using matrices:\[\begin{pmatrix} \sum x_i^2 & \sum x_i \\sum x_i & n \end{pmatrix} \begin{pmatrix} m \b \end{pmatrix} = \begin{pmatrix} \sum x_i y_i \\sum y_i \end{pmatrix}\]In this setup:

• The first matrix is called the coefficient matrix.
• The second is the vector of unknowns \(m\) and \(b\).
• The third is the result vector from our equations.

We solve for \(m\) and \(b\) by inverting the coefficient matrix and multiplying both sides of the equation. This method provides an efficient way to find solutions, especially when dealing with large sets of data. Moreover, it better illustrates the linear algebra principles underlying the least squares regression process.