Problem 19 - An Introduction to Mathematical Statistics and Its Applications

Question

Set up (but do not solve) the equations necessary to determine the least squares estimates for the trigonometric model, $$ y=a+b x+c \sin x $$ Assume that the data consist of the random sample \(\left(x_{1},\right.\), \(\left.y_{1}\right),\left(x_{2}, y_{2}\right), \ldots\), and \(\left(x_{n}, y_{n}\right)\).

Step-by-Step Solution

Answer

The system of equations to find the least squares estimates of a, b and c in the trigometric model \( y = a + b x + c \sin(x) \) are given by: \[ n a + (\sum_{i=1}^{n} x_i) b + (\sum_{i=1}^{n}\sin x_i) c = \sum_{i=1}^{n} y_{i} \], \[ (\sum_{i=1}^{n} x_i) a + (\sum_{i=1}^{n} x_{i}^2) b + (\sum_{i=1}^{n} x_{i} \sin x_{i}) c = \sum_{i=1}^{n} x_{i} y_{i} \], \[ (\sum_{i=1}^{n}\sin x_{i}) a + (\sum_{i=1}^{n} x_{i}\sin x_{i}) b + (\sum_{i=1}^{n} \sin^{2} x_{i}) c = \sum_{i=1}^{n} y_{i}\sin x_{i} \]. These equations can be solved to find the parameters a, b and c.

Step 1: Define the Residual Sum of Squares (RSS)

The residuals, or differences between the observed (y) and estimated (\( \hat{y} \)) values, are summed and squared in the RSS. The model’s equation can be used to calculate \( \hat{y} = a+b x_{i} + c \sin x_{i} \), so the residuals for each observation (\( e_{i} \)) = \( y_{i} - \hat{y} \) = \( y_{i} - a - b x_{i} - c \sin x_{i} \). RSS = \( \sum_{i=1}^{n} e_{i}^{2} \) = \( \sum_{i=1}^{n} (y_{i} - a - b x_{i} - c \sin x_{i})^{2} \).

Step 2: Partial Differentiation w.r.t a, b, c

To minimize the RSS, the first-order condition requires that the gradient of the RSS function equals zero. This is accomplished by setting the first partial derivatives of the RSS function with respect to a, b, and c equal to zero. \[ \frac{\partial RSS}{\partial a} = -2 \sum_{i=1}^{n} (y_{i} - a - b x_{i} - c \sin x_{i}) = 0 \] \[ \frac{\partial RSS}{\partial b} = -2 \sum_{i=1}^{n} x_{i} (y_{i} - a - b x_{i} - c \sin x_{i}) = 0 \] \[ \frac{\partial RSS}{\partial c} = -2 \sum_{i=1}^{n} \sin x_{i} (y_{i} - a - b x_{i} - c \sin x_{i}) = 0 \]

Step 3: Set Up the System of Equations

The result of the previous step is a system of linear equations, known as the normal equations, which can be rearranged to solve the parameters a, b, and c. \[ n a + (\sum_{i=1}^{n} x_i) b + (\sum_{i=1}^{n}\sin x_i) c = \sum_{i=1}^{n} y_{i} \] \[ (\sum_{i=1}^{n} x_i) a + (\sum_{i=1}^{n} x_{i}^2) b + (\sum_{i=1}^{n} x_{i} \sin x_{i}) c = \sum_{i=1}^{n} x_{i} y_{i} \] \[ (\sum_{i=1}^{n}\sin x_{i}) a + (\sum_{i=1}^{n} x_{i}\sin x_{i}) b + (\sum_{i=1}^{n} \sin^{2} x_{i}) c = \sum_{i=1}^{n} y_{i}\sin x_{i} \]

Key Concepts

Residual Sum of Squares

In the world of least squares estimation, the Residual Sum of Squares (RSS) plays a crucial role. It's a measure that helps us understand how well our model fits the observed data. Think of RSS as the total distance between the data points and the predicted values from your model. The smaller this number, the better the model’s predictions align with the actual observations.
To obtain the RSS for a trigonometric model like \( y=a+b x+c \sin x \), we first calculate the predicted values (\( \hat{y} \)) using the parameters \( a, b, \) and \( c \). For each data point \(( x_i, y_i)\), we compute the residual, \( e_i = y_i - \hat{y} = y_i - a - b x_i - c \sin x_i \). We square each residual and sum them all up to get the RSS: \( \text{RSS} = \sum_{i=1}^{n} (y_i - a - b x_i - c \sin x_i)^2 \).
This formula reflects the overall discrepancy between the actual and estimated values. The objective of least squares is to minimize this discrepancy.

Normal Equations

Normal Equations are central to solving the least squares problem, particularly when dealing with linear equations. When we perform least squares regression, we're trying to find the parameter values that minimize the RSS. To achieve this, we use the first-order conditions that result in these so-called "Normal Equations."
After applying partial differentiation to RSS with respect to the parameters \(a, b,\) and \(c\), we derive a system of equations. These equations, when solved simultaneously, will provide us with the estimates for \( a, b, \) and \( c \).
These are the normal equations:

• \( n a + (\sum_{i=1}^{n} x_i) b + (\sum_{i=1}^{n}\sin x_i) c = \sum_{i=1}^{n} y_{i} \)
• \( (\sum_{i=1}^{n} x_i) a + (\sum_{i=1}^{n} x_{i}^2) b + (\sum_{i=1}^{n} x_{i} \sin x_{i}) c = \sum_{i=1}^{n} x_{i} y_{i} \)
• \( (\sum_{i=1}^{n}\sin x_{i}) a + (\sum_{i=1}^{n} x_{i}\sin x_{i}) b + (\sum_{i=1}^{n} \sin^{2} x_{i}) c = \sum_{i=1}^{n} y_{i}\sin x_{i} \)

By solving these equations, we can find the values of \( a, b,\) and \( c \) that provide the best fit for the model.

Partial Differentiation

Partial differentiation is a technique used to find the rate at which a function changes as its variables change. When minimizing the Residual Sum of Squares in least squares regression, partial differentiation helps us determine the best parameter values for our model.
In the context of least squares, we take the partial derivative of the RSS with respect to each parameter involved in the model—here, \( a, b,\) and \( c \). By setting these derivatives equal to zero, we locate the minima where the RSS is smallest. This is an essential step because it ensures that the prediction errors are as minimized as possible across the dataset.
This process results in equations that form what we call the Normal Equations. These equations are derived from setting:

• \( \frac{\partial \text{RSS}}{\partial a} = -2 \sum_{i=1}^{n} (y_{i} - a - b x_{i} - c \sin x_{i}) = 0 \)
• \( \frac{\partial \text{RSS}}{\partial b} = -2 \sum_{i=1}^{n} x_{i} (y_{i} - a - b x_{i} - c \sin x_{i}) = 0 \)
• \( \frac{\partial \text{RSS}}{\partial c} = -2 \sum_{i=1}^{n} \sin x_{i} (y_{i} - a - b x_{i} - c \sin x_{i}) = 0 \)

Partial differentiation is key to systematically handling each variable and solving these equations to achieve an optimal model solution.