Problem 24 - Applied Linear Statistical Models

Question

(Calculus needed.) Consider the multiple regression model: $$Y_{i}=\beta_{0}+\beta_{1} X_{i 1}+\beta_{2} X_{i 1}^{2}+\beta_{3} X_{i 2}+\varepsilon_{i} \quad i=1, \ldots, n$$ where the \(\varepsilon_{i}\) are independent \(N\left(0, \sigma^{2}\right)\) a. State the least squares criterion and derive the least squares normal equations. b. State the likelihood function and explain why the maximum likelihood estimators will be the same as the least squares estimators.

Step-by-Step Solution

Answer

The least squares estimators and maximum likelihood estimators are the same, both derived from minimizing the sum of squared residuals and the negative log-likelihood function respectively.

Step 1: State the Least Squares Criterion

The least squares criterion minimizes the sum of the squared residuals. The residual for each observation is the difference between the observed value and the predicted value. The objective is to find the parameters \(\beta_0, \beta_1, \beta_2, \beta_3\) that minimize the sum: \[ S(\beta_0, \beta_1, \beta_2, \beta_3) = \sum_{i=1}^{n} ( Y_{i} - (\beta_0 + \beta_1 X_{i1} + \beta_2 X_{i1}^{2} + \beta_3 X_{i2}) )^2. \]

Step 2: Derive the Normal Equations

To derive the normal equations, take the partial derivatives of the sum of squares function with respect to each parameter, set them equal to zero, and solve. This yields the normal equations: \[ \frac{\partial S}{\partial \beta_0} = -2 \sum_{i=1}^{n} (Y_{i} - \beta_{0} - \beta_{1} X_{i1} - \beta_{2} X_{i1}^{2} - \beta_{3} X_{i2}) = 0, \] \[ \frac{\partial S}{\partial \beta_1} = -2 \sum_{i=1}^{n} X_{i1}(Y_{i} - \beta_{0} - \beta_{1} X_{i1} - \beta_{2} X_{i1}^{2} - \beta_{3} X_{i2}) = 0, \] \[ \frac{\partial S}{\partial \beta_{2}} = -2 \sum_{i=1}^n X_{i1}^{2}(Y_{i} - \beta_{0} - \beta_{1} X_{i1} - \beta_{2} X_{i1}^{2} - \beta_{3} X_{i2}) = 0, \] \[ \frac{\partial S}{\partial \beta_{3}} = -2 \sum_{i=1}^n X_{i2}(Y_{i} - \beta_{0} - \beta_{1} X_{i1} - \beta_{2} X_{i1}^{2} - \beta_{3} X_{i2}) = 0. \] These equations form a system of linear equations that can be solved for \(\beta_0, \beta_1, \beta_2, \beta_3\).

Step 3: State the Likelihood Function

For the given model where errors \(\varepsilon_i\) are normally distributed \(N(0,\sigma^2)\), the likelihood function is: \[ L(\beta_0, \beta_1, \beta_2, \beta_3, \sigma^2) = \prod_{i=1}^{n} \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left( -\frac{(Y_i - \beta_0 - \beta_1 X_{i1} - \beta_2 X_{i1}^2 - \beta_3 X_{i2})^2}{2\sigma^2} \right). \]

Step 4: Explain Maximum Likelihood Estimators

To find the maximum likelihood estimators (MLEs), we maximize the likelihood function with respect to the parameters or equivalently minimize the negative log-likelihood. Given the likelihood function, the negative log-likelihood for this model is: \[ -\log L(\beta_0, \beta_1, \beta_2, \beta_3, \sigma^2) = \frac{n}{2} \log(2\pi) + \frac{n}{2} \log(\sigma^2) + \frac{1}{2\sigma^2} \sum_{i=1}^{n} (Y_i - \beta_0 - \beta_1 X_{i1} - \beta_2 X_{i1}^2 - \beta_3 X_{i2})^2. \] Minimizing this negative log-likelihood leads to the same normal equations derived in Step 2, which means the MLEs for \(\beta_0, \beta_1, \beta_2, \beta_3\) are identical to the least squares estimators.

Key Concepts

Least Squares Criterion

The least squares criterion is a fundamental method used in regression analysis. It helps us find the optimal parameters for our model by minimizing the sum of the squared differences between the observed values and the values predicted by the model. These differences, called residuals, measure how well our model fits the data. Specifically, in our multiple regression model, we aim to minimize the sum:
\[ S(\beta_0, \beta_1, \beta_2, \beta_3) = \sum_{i=1}^{n} ( Y_{i} - (\beta_0 + \beta_1 X_{i1} + \beta_2 X_{i1}^{2} + \beta_3 X_{i2}) )^2. \]
This equation tells us that we must adjust our parameters \( \beta_0, \beta_1, \beta_2, \beta_3 \) to make the sum of these squared differences as small as possible.

Normal Equations

Once we have defined the least squares criterion, the next step is to derive the normal equations. These equations help us find the optimal parameters by solving a system of linear equations. To derive them, we calculate the partial derivatives of the sum of squares function with respect to each parameter and set these derivatives equal to zero. This process yields the following equations:
\[ -2 \sum_{i=1}^{n} (Y_{i} - \beta_{0} - \beta_{1} X_{i1} - \beta_{2} X_{i1}^{2} - \beta_{3} X_{i2}) = 0, \]
\[ -2 \sum_{i=1}^{n} X_{i1}(Y_{i} - \beta_{0} - \beta_{1} X_{i1} - \beta_{2} X_{i1}^{2} - \beta_{3} X_{i2}) = 0, \]
\[ -2 \sum_{i=1}^n X_{i1}^{2}(Y_{i} - \beta_{0} - \beta_{1} X_{i1} - \beta_{2} X_{i1}^{2} - \beta_{3} X_{i2}) = 0, \]
\[ -2 \sum_{i=1}^n X_{i2}(Y_{i} - \beta_{0} - \beta_{1} X_{i1} - \beta_{2} X_{i1}^{2} - \beta_{3} X_{i2}) = 0. \]
By solving this system of linear equations, we can determine the values of \( \beta_0, \beta_1, \beta_2, \beta_3 \) that minimize the sum of squared residuals.

Likelihood Function

The likelihood function is a key concept in statistical modeling, used to estimate the parameters of a model. For our regression model, where the errors \( \varepsilon_i \) are normally distributed with mean zero and variance \( \sigma^2 \), the likelihood function is given by:
\[ L(\beta_0, \beta_1, \beta_2, \beta_3, \sigma^2) = \prod_{i=1}^{n} \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left( -\frac{(Y_i - \beta_0 - \beta_1 X_{i1} - \beta_2 X_{i1}^2 - \beta_3 X_{i2})^2}{2\sigma^2} \right). \]
This function represents the probability of observing the given data, assuming the model parameters are correct. By maximizing this function, we find the most likely values of \( \beta_0, \beta_1, \beta_2, \beta_3 \) and \( \sigma^2 \).

Maximum Likelihood Estimators

To derive the maximum likelihood estimators (MLEs), we seek the parameters that maximize the likelihood function. In practice, we usually minimize the negative log-likelihood because it is easier to work with. For our model, the negative log-likelihood is:
\[ -\log L(\beta_0, \beta_1, \beta_2, \beta_3, \sigma^2) = \frac{n}{2} \log(2\pi) + \frac{n}{2} \log(\sigma^2) + \frac{1}{2\sigma^2} \sum_{i=1}^{n} (Y_i - \beta_0 - \beta_1 X_{i1} - \beta_2 X_{i1}^2 - \beta_3 X_{i2})^2. \]
When we minimize this expression, we obtain the same normal equations we derived earlier using the least squares criterion. Therefore, the MLEs for \( \beta_0, \beta_1, \beta_2, \beta_3 \) are identical to the least squares estimators, confirming that both methods yield the same results.