Problem 1 - A Modern Introduction to Probability and Statistics: Understanding Why and How

Question

Suppose our dataset is a realization of a random sample \(X_{1}, X_{2}, \ldots, X_{n}\) from a uniform distribution on the interval \([-\theta, \theta]\), where \(\theta\) is unknown. a. Show that $$ T=\frac{3}{n}\left(X_{1}^{2}+X_{2}^{2}+\cdots+X_{n}^{2}\right) $$ is an unbiased estimator for \(\theta^{2}\). b. Is \(\sqrt{T}\) also an unbiased estimator for \(\theta\) ? If not, argue whether it has positive or negative bias.

Step-by-Step Solution

Answer

a: Yes, \( T \) is unbiased for \( \theta^2 \). b: No, \( \sqrt{T} \) is biased with a negative bias.

Step 1: Understanding Bias of an Estimator

An estimator \( T \) is called unbiased for a parameter \( \theta \) if the expected value \( E[T] = \theta \). For \( T \) to be an unbiased estimator of \( \theta^2 \), we need to show that the expected value of \( T \) is equal to \( \theta^2 \).

Step 2: Compute Expected Value of Original Function

We are given that \( X_1, X_2, \ldots, X_n \sim U[-\theta, \theta] \). The variance of a uniform distribution \( U[a, b] \) is given by \( \frac{(b - a)^2}{12} \). Here, \( a = -\theta \) and \( b = \theta \), so \[\text{Var}(X_i) = \frac{(\theta - (-\theta))^2}{12} = \frac{4\theta^2}{12} = \frac{\theta^2}{3}. \]The expected value \( E[X_i^2] \) is \( \theta^2/3 \) since \( \mathbb{E}[X_i^2] = \text{Var}(X_i) + (\mathbb{E}[X_i])^2 \), where \( \mathbb{E}[X_i] = 0 \) for the symmetric distribution. Thus, \[E[X_i^2] = \frac{\theta^2}{3}.\]

Step 3: Show Unbiasedness of T

Substituting \( E[X_i^2] = \frac{\theta^2}{3} \) into the formula for \( T \), we have \[E\left[\frac{3}{n}\left(X_1^2 + X_2^2 + \cdots + X_n^2\right)\right] = \frac{3}{n} \times n \times \frac{\theta^2}{3} = \theta^2.\]This confirms that \( T \) is indeed an unbiased estimator of \( \theta^2 \).

Step 4: Assess \\sqrt{T} for Unbiasedness

To check if \( \sqrt{T} \) is an unbiased estimator for \( \theta \), consider how transformations affect bias. Since the square root function is not linear, \[E[\sqrt{T}] eq \sqrt{E[T]} = \sqrt{\theta^2} = \theta. \]The expectation of \( \sqrt{T} \) would generally not equal \( \theta \), meaning it is not an unbiased estimator.

Step 5: Determine Bias Direction of \\sqrt{T}

The bias of \( \sqrt{T} \) comes from Jensen's inequality, which states that for a convex function (like \( x^2 \)), \( E[f(X)] \geq f(E[X]) \). Its inverse, the concave function \( \sqrt{x} \), implies \( E[\sqrt{T}] \leq \sqrt{E[T]} \). Thus, \( \sqrt{T} \) is biased downwards, meaning the bias is negative.

Key Concepts

Uniform Distribution

The uniform distribution is a type of probability distribution in which all outcomes are equally likely within a given interval. If you imagine a fair die, each side has an equal chance of showing up. In a continuous sense, this means that every number within a specified range has an equal probability of being observed. In mathematical terms:

• The uniform distribution on the interval a\ a, b\ ba (denoted as \( U[a, b] \)) assigns equal probability density to every point between \( a \) and \( b \).
• The probability density function (PDF) for a uniform distribution \( U[a, b] \) is given by \( \frac{1}{b-a} \) for \( x \) within \( [a, b] \).

In our context, the random sample is drawn from a uniform distribution \([-\theta, \theta]\). It means that each outcome between \(-\theta\) and \(\theta\) has an equal chance of occurring. This uniformity plays a key role in determining statistics like mean and variance, which are crucial for creating estimators.

Variance

Variance is a measure of how much a set of numbers, such as a sample, spreads out from their mean. If the numbers are close to each other, the variance is small. If they are spread out over a wider range, the variance is larger.

• It gives us insights into the data's diversity and helps in estimating the degree of uncertainty involved.
• The formula for variance of a random variable \( X \) is given as \( \text{Var}(X) = E[X^2] - (E[X])^2 \).

For a uniform distribution like \( U[-\theta, \theta] \), the variance is calculated using:\[ \text{Var}(X) = \frac{(b - a)^2}{12} = \frac{(\theta - (-\theta))^2}{12} = \frac{4\theta^2}{12} = \frac{\theta^2}{3}. \]Understanding variance is key when considering the expected value of \( X_i^2 \), since \( E[X_i^2] = \text{Var}(X_i) + (E[X_i])^2 \), assisting in determining how accurately \( T \) estimates \( \theta^2 \).

Random Sample

A random sample is a subset of a statistical population in which each member has an equal chance of being chosen. This method is used to make inferences about the entire population without having to survey every individual. When we draw such a sample from a uniform distribution, we're essentially picking numbers randomly but with ruled uniformity.

• In statistical terms, each \( X_i \) in a random sample \( X_1, X_2, \ldots, X_n \) from \( U[-\theta, \theta] \) comes from the same distribution.
• This process assumes independence and identical distribution for all \( X_i \), which is crucial for valid estimator derivation.

Random sampling ensures that the results of statistical analyses are free from bias. Such assumptions are integral to creating unbiased estimators, as it allows for consistent calculation of expected values that support inference.

Jensen's Inequality

Jensen's inequality is a vital concept in probability and statistics, indicating that for a convex function \( f \), the expectation \( E[f(X)] \geq f(E[X]) \). In simpler terms, it provides insight into how transformations of random variables preserve or alter mean values when passing through non-linear functions.

• If \( f \) is a convex function, then the mean of \( f(X) \) will be greater than or equal to the value obtained by putting the mean into \( f \).
• If \( f \) is a concave function, as it is with \( \sqrt{x} \), the inequality flips: \( E[\sqrt{T}] \leq \sqrt{E[T]} \).

In this exercise, Jensen's inequality helps us understand why transforming \( T \) with a square root results in a biased estimator, \( \sqrt{T} \), for \( \theta \). We see that due to the concavity of \( \sqrt{x} \), \( \sqrt{T} \) underestimates \( \theta \), resulting in a negative bias.