Problem 11 - A Modern Introduction to Probability and Statistics: Understanding Why and How

Question

\(\square\) In Exercise \(17.9\) we modeled diameters of black cherry trees with the linear regression model (without intercept) $$ Y_{i}=\beta x_{i}+U_{i} $$ for \(i=1,2, \ldots, n\). As usual, the \(U_{i}\) here are independent random variables with \(\mathrm{E}\left[U_{i}\right]=0\), and \(\operatorname{Var}\left(U_{i}\right)=\sigma^{2}\). We considered three estimators for the slope \(\beta\) of the line \(y=\beta x\) the socalled least squares estimator \(T_{1}\) (which will be considered in Chapter 22), the average slope estimator \(T_{2}\), and the slope of the averages estimator \(T_{3}\). These estimators are defined by: $$ T_{1}=\frac{\sum_{i=1}^{n} x_{i} Y_{i}}{\sum_{i=1}^{n} x_{i}^{2}}, \quad T_{2}=\frac{1}{n} \sum_{i=1}^{n} \frac{Y_{i}}{x_{i}}, \quad T_{3}=\frac{\sum_{i=1}^{n} Y_{i}}{\sum_{i=1}^{n} x_{i}} . $$ In Exercise \(19.8\) it was shown that all three estimators are unbiased. Compute the MSE of all three estimators. Remark: it can be shown that \(T_{1}\) is always more efficient than \(T_{3}\), which in turn is more efficient than \(T_{2}\). To prove the first inequality one uses a famous inequality called the Cauchy Schwartz inequality; for the second inequality one uses Jensen's inequality (can you see how?).

Step-by-Step Solution

Answer

The MSE for \(T_1\), \(T_2\), and \(T_3\) are their variances: \(\frac{\sigma^2}{\sum x_i^2}\), \(\frac{\sigma^2}{n^2} \sum \frac{1}{x_i^2}\), and \(\frac{\sigma^2 n}{(\sum x_i)^2}\), respectively.

Step 1: Understand the Task

We need to compute the Mean Squared Error (MSE) of the three different estimators of the slope \(\beta\) in a linear regression model without intercept. These estimators are \(T_1\), \(T_2\), and \(T_3\). The MSE is defined as \( \text{MSE}(T) = \text{Var}(T) + [\text{Bias}(T)]^2 \). Since all estimators are unbiased, the task simplifies to calculating the variance of each estimator.

Step 2: Determine the Variance of T1

The estimator \(T_1\) is given by \(T_1 = \frac{\sum_{i=1}^{n} x_{i} Y_{i}}{\sum_{i=1}^{n} x_{i}^{2}}\). Since \( \text{E}[Y_i] = \beta x_i \) and \( Y_i = \beta x_i + U_i \), the variance of \(T_1\) can be computed as \( \text{Var}(T_1) = \frac{\sigma^2}{\sum_{i=1}^n x_i^2} \), as shown by plugging in the definition of \(Y_i\) and using properties of variance.

Step 3: Determine the Variance of T2

The estimator \(T_2\) is given by \(T_2 = \frac{1}{n} \sum_{i=1}^{n} \frac{Y_i}{x_i}\). The variance for each term \( \frac{Y_i}{x_i} \) is \( \text{Var}\left(\frac{U_i}{x_i}\right) = \frac{\sigma^2}{x_i^2} \). Therefore, \( \text{Var}(T_2) = \frac{\sigma^2}{n^2} \sum_{i=1}^{n} \frac{1}{x_i^2} \).

Step 4: Determine the Variance of T3

The estimator \(T_3\) is given by \(T_3 = \frac{\sum_{i=1}^{n} Y_{i}}{\sum_{i=1}^{n} x_{i}}\). The variance for this estimator can be found as \( \text{Var}(T_3) = \frac{\sigma^2 n}{\left(\sum_{i=1}^{n} x_i\right)^2} \), stemming from the aggregation of independent terms \(U_i\) over the sums.

Key Concepts

Understanding Linear Regression

Linear regression is a basic yet powerful statistical technique used to model the relationship between a dependent variable and one or more independent variables. In the scenario we are examining, we are using simple linear regression without an intercept, specifically focusing on the diameters of black cherry trees.
In this context, the mathematical formula becomes:

• \( Y_{i} = \beta x_{i} + U_{i} \), where \(Y_i\) is the observed diameter, \(x_i\) is the predictor variable, \(\beta\) is the slope of the line, and \(U_i\) is an error term.

The main goal of linear regression is to find the best-fitting line through the data points. This is accomplished by estimating the slope \(\beta\), which indicates how much \(Y\) changes for a one-unit change in \(x\).
Multiple methods or estimators can be used to find this \(\beta\). In our exercise, the primary interest is in comparing three estimators: \(T_1\), \(T_2\), and \(T_3\), and assessing their efficiency using the Mean Squared Error (MSE). Understanding the linear regression concept sets a solid foundation for diving deeper into this and appreciating the role of each estimator.

Insight into Unbiased Estimators

A crucial goal in statistics is to use unbiased estimators when estimating parameters like \(\beta\) in linear regression. An estimator is unbiased if the expected value of the estimator equals the true parameter value. This means the estimator is accurate on average.
For our problem, all three estimators, \(T_1\), \(T_2\), and \(T_3\), are unbiased. This simplifies the Mean Squared Error (MSE) calculation because the bias component equals zero, reducing the task to merely calculating the variance.
Unbiased estimators are especially favored in statistical modeling because they do not systematically overestimate or underestimate the true parameter. However, being unbiased doesn’t automatically imply that an estimator is the best. Other properties like variance need to be considered, hence the importance of MSE in comparing them. The variance measures how much the estimator's values spread around the expected value. Since these estimators are unbiased, a lower variance leads to a lower MSE, indicating a more reliable estimator in terms of precision.

Applying the Cauchy-Schwarz Inequality

The Cauchy-Schwarz inequality is a mathematical principle that is highly applicable in various fields, including statistics. It states that for any vectors \(a\) and \(b\), the absolute value of their dot product is at most the product of their magnitudes.
In the context of our exercise, this inequality can prove why certain estimators might be more efficient than others. Specifically, it helps demonstrate why the least squares estimator \(T_1\) is always more efficient than the slope of averages estimator \(T_3\).
Efficiency here means having a lower variance, which translates to a more precise estimate of \(\beta\). The Cauchy-Schwarz inequality aids in these comparisons by setting a theoretical bound on the product of sums involving the data, which in turn relates to the variance calculations. By providing a constraint, this inequality helps clarify why \(T_1\)'s formulation inherently leads to smaller variance than \(T_3\), making it the preferred estimator when precision is critical. Understanding this concept is essential for anyone delving deep into statistical estimations and optimizations.