Problem 6 - Applied Linear Regression

Question

Let \(\mathbf{U}\) be an \(n \times 1\) vector with 1 as its first element and 0 s elsewhere. Consider computing the regression of \(\mathbf{U}\) on an \(n \times p^{\prime}\) full rank matrix \(\mathbf{X}\) As usual, let \(\mathbf{H}=\mathbf{X}\left(\mathbf{X}^{\prime} \mathbf{X}\right)^{-1} \mathbf{X}^{\prime}\) be the hat matrix with elements \(h_{i j}\) a. Show that the elements of the vector of fitted values from the regression of \(\mathbf{U}\) on \(\mathbf{X}\) are the \(h_{1 j}, j=1,2, \ldots, n\) b. Show that the first element of the vector of residuals is \(1-h_{11},\) and the other elements are \(-h_{1 j}, j>1\)

Step-by-Step Solution

Answer

In summary, for the regression of the vector \(\mathbf{U}\) on the matrix \(\mathbf{X}\), the vector of fitted values is given by elements \(h_{1j}, j=1,2, \ldots, n\), where \(h_{ij}\) are the elements of the hat matrix \(\mathbf{H}\). The first element of the residuals vector is \(1-h_{11}\), and the other elements are \(-h_{1j}, j>1\).

Step 1: Define the fitted values vector and residuals vector

The fitted values vector, denoted by \(\mathbf{\hat{U}}\), comes from the regression of \(\mathbf{U}\) on \(\mathbf{X}\). The residuals vector, denoted by \(\mathbf{\epsilon}\), is the difference between the observed vector \(\mathbf{U}\) and the fitted values vector \(\mathbf{\hat{U}}\). Thus, we have: \[ \mathbf{\hat{U}} = \mathbf{H} \mathbf{U} = \mathbf{X}(\mathbf{X}^{\prime} \mathbf{X})^{-1} \mathbf{X}^{\prime} \mathbf{U} \] and \[ \mathbf{\epsilon} = \mathbf{U} - \mathbf{\hat{U}} \]

Step 2: Compute the fitted values vector \(\mathbf{\hat{U}}\)

To find the fitted values vector, we will multiply the hat matrix \(\mathbf{H}\) by the \(\mathbf{U}\) vector. Since \(\mathbf{U}\) has only one non-zero element (its first element is 1), we can directly multiply the first column of the hat matrix to this element: \[ \mathbf{\hat{U}}_j = h_{1j} \cdot 1, \quad j = 1, 2, \ldots, n \] Thus, the elements of the vector of fitted values are the \(h_{1j}, j=1,2, \ldots, n\).

Step 3: Compute the residuals vector \(\mathbf{\epsilon}\)

The residuals vector is the difference between the observed vector \(\mathbf{U}\) and the fitted values vector \(\mathbf{\hat{U}}\): \[ \mathbf{\epsilon}_j = \mathbf{U}_j - \mathbf{\hat{U}}_j \] For the first element, we have: \[ \mathbf{\epsilon}_1 = \mathbf{U}_1 - \mathbf{\hat{U}}_1 = 1 - h_{11} \] For the other elements, \(j > 1\), we have \(\mathbf{U}_j = 0\), so: \[ \mathbf{\epsilon}_j = 0 - h_{1j} = -h_{1j}, \quad j>1 \] Therefore, the first element of the vector of residuals is \(1-h_{11}\), and the other elements are \(-h_{1j}, j>1\).

Key Concepts

Hat Matrix in Linear Regression

In linear regression, the hat matrix, denoted as \( \mathbf{H} \), plays an essential role in predicting the fitted values from the model. It transforms the original values into the predicted values by projecting them onto the space spanned by the explanatory variables.

The formula for the hat matrix is \[ \mathbf{H} = \mathbf{X}(\mathbf{X}^\prime \mathbf{X})^{-1} \mathbf{X}^\prime \. \] This matrix is called the hat matrix because it 'puts a hat' on \( \mathbf{Y} \), which is the vector of dependent variables, to get the fitted values \( \mathbf{\hat{Y}} \).

The hat matrix can be used to determine how much influence each observation has on its fitted value. The diagonal elements \( h_{ii} \) show the leverage of each observation, indicating how far it is from the center of the explanatory variable space.

Fitted Values Vector

The fitted values vector, often symbolized as \( \mathbf{\hat{U}} \), represents the estimated values obtained by a linear regression model. These values are predicted from the regression equation and the observed data points.

Using the hat matrix, the fitted values vector for a vector \( \mathbf{U} \) when regressed on a full rank matrix \( \mathbf{X} \) is given by \[ \mathbf{\hat{U}} = \mathbf{H} \mathbf{U} \. \] The elements of \( \mathbf{\hat{U}} \) correspond to the predicted values for each individual observation in the data set. In our specific case, the prediction for the j-th observation is simply \( h_{1j} \) since that's the component interacting with the non-zero element of \( \mathbf{U} \).

Residuals Vector

The residuals vector \( \mathbf{\epsilon} \) illustrates the discrepancies between the observed values and the values predicted by the model. Essentially, residuals are the errors of the model's predictions. For a more formal definition, we have

\[ \mathbf{\epsilon} = \mathbf{U} - \mathbf{\hat{U}} = \mathbf{U} - \mathbf{H} \mathbf{U} \. \] In the task at hand, the first element of the residuals vector, \( \mathbf{\epsilon}_1 \), represents the error for the first observation and is calculated as \( 1 - h_{11} \). All other elements, for \( j > 1 \), are the negation of the corresponding elements \( h_{1j} \) in the hat matrix, signifying their respective prediction errors.

Full Rank Matrix

A full rank matrix is a fundamental concept in linear regression, as it assures the uniqueness of solutions in the regression analysis. A matrix \( \mathbf{X} \) is said to have full rank if its columns are linearly independent, meaning no column can be written as a linear combination of the others.

This property is crucial because the matrix \( \mathbf{X}^\prime \mathbf{X} \) must be invertible for the hat matrix \( \mathbf{H} \) to be calculated. If the matrix \( \mathbf{X} \) does not have full rank, it would indicate multicollinearity among the explanatory variables, which can lead to problems like inflated variances of the parameter estimates. The matrix we consider in the exercise is a full rank matrix, thereby guaranteeing a well-defined hat matrix and a stable linear regression solution.