Problem 12 - A Modern Introduction to Probability and Statistics: Understanding Why and How

Question

Let \(X_{1}, X_{2}, \ldots\) be a sequence of independent random variables, with \(\mathrm{E}\left[X_{i}\right]=\mu_{i}\) and \(\operatorname{Var}\left(X_{i}\right)=\) \(\sigma_{i}^{2}\), for \(i=1,2, \ldots\) Suppose that \(0<\sigma_{i}^{2} \leq M\), for all \(i .\) Let \(a\) be an arbitrary positive number. a. Apply Chebyshev's inequality to show that $$ \mathrm{P}\left(\left|\bar{X}_{n}-\frac{1}{n} \sum_{i=1}^{n} \mu_{i}\right|>a\right) \leq \frac{\operatorname{Var}\left(X_{1}\right)+\cdots+\operatorname{Var}\left(X_{n}\right)}{n^{2} a^{2}} $$ b. Conclude from a that $$ \lim _{n \rightarrow \infty} \mathrm{P}\left(\left|\bar{X}_{n}-\frac{1}{n} \sum_{i=1}^{n} \mu_{i}\right|>a\right)=0 $$ Check that the law of large numbers is a special case of this result.

Step-by-Step Solution

Answer

a. Use Chebyshev's inequality with variance bound; b. As \( n \to \infty \), probability goes to 0, confirming LLN.

Step 1: Define the Sequence and Sample Mean

Let \( X_1, X_2, \ldots, X_n \) be independent random variables with expected values \( \mu_i \) and variances \( \sigma_i^2 \). We define the sample mean \( \bar{X}_n = \frac{1}{n} \sum_{i=1}^{n} X_i \). We are interested in the probability \( \mathrm{P}\left( \left| \bar{X}_n - \frac{1}{n} \sum_{i=1}^{n} \mu_i \right| > a \right) \).

Step 2: Use Chebyshev's Inequality

Chebyshev's inequality states that for any random variable \( Y \) with mean \( \mu \) and variance \( \sigma^2 \), \( \mathrm{P}(|Y - \mu| > k) \leq \frac{\sigma^2}{k^2} \). In our setup, set \( Y = \bar{X}_n \), \( \mu = \frac{1}{n} \sum_{i=1}^{n} \mu_i \), and \( k = a \). The variance of \( \bar{X}_n \) is \( \operatorname{Var}(\bar{X}_n) = \frac{1}{n^2} \sum_{i=1}^{n} \sigma_i^2 \), thus Chebyshev's inequality gives us:\[ \mathrm{P}\left( \left| \bar{X}_n - \frac{1}{n} \sum_{i=1}^{n} \mu_i \right| > a \right) \leq \frac{\frac{1}{n^2} \sum_{i=1}^{n} \sigma_i^2}{a^2} = \frac{\operatorname{Var}(X_1) + \cdots + \operatorname{Var}(X_n)}{n^2 a^2} \].

Step 3: Evaluate the Limit

From the inequality in Step 2, note that since \( 0 < \sigma_i^2 \leq M \), we have \( \sum_{i=1}^{n} \sigma_i^2 \leq nM \). Divide both sides by \( n^2 a^2 \):\[ \frac{\sum_{i=1}^{n} \sigma_i^2}{n^2 a^2} \leq \frac{nM}{n^2 a^2} = \frac{M}{n a^2} \].As \( n \to \infty \), \( \frac{M}{n a^2} \to 0 \). Thus,\[ \lim_{n \to \infty} \mathrm{P}\left( \left| \bar{X}_n - \frac{1}{n} \sum_{i=1}^{n} \mu_i \right| > a \right) = 0 \].

Step 4: Connection to the Law of Large Numbers

The result obtained shows that as \( n \to \infty \), the sample mean \( \bar{X}_n \) converges in probability to the average of expectations \( \frac{1}{n} \sum_{i=1}^{n} \mu_i \). In the special case where \( \mu_1 = \mu_2 = \cdots = \mu \) and \( \sigma_i^2 \leq M \), this becomes the classical weak law of large numbers, stating that \( \bar{X}_n \to \mu \) in probability.

Key Concepts

Law of Large Numbers

The law of large numbers is a fundamental theorem in probability and statistics. It explains how, as the size of a sample increases, the sample mean gets closer to the expected value or average for that population.
This concept is crucial because it assures that we can estimate the true expected value of a population by using a sufficiently large and random sample. The law comes in two forms: weak and strong.

• Weak Law: States that the sample mean converges in probability towards the expected value as the number of observations grows.
• Strong Law: States that the sample mean almost surely converges to the expected value as the sample size increases indefinitely.

In the context of our problem, the weak law is validated. When the variances are bounded and the sample size increases, the probability of the sample mean deviating from its expected value by more than any small positive number approaches zero.

Independent Random Variables

Random variables are independent if the occurrence or realization of one has no effect on the probability of occurrence of the other. This means that knowing the value of one does not change the probability distributions of the others.
In our exercise, each random variable in the sequence is independent, which makes it possible to apply the addition rule of variances. Specifically, since the variables are independent:

• The variance of the sum of the random variables is the sum of their variances.
• This property simplifes the process of finding the variance of the sample mean, enabling us to apply Chebyshev's inequality.

Independence is a strong assumption but is quite commonly used, allowing simpler modeling of complex real-world processes.

Sample Mean Convergence

Sample mean convergence refers to how the calculated average from a sample approaches the actual mean of the population as the sample size increases.
In the exercise, we use Chebyshev's inequality to illustrate that the variance of the sample mean decreases as more data is collected, resulting in tighter confidence intervals around the population mean.

The convergence described is in probability, a notion formalized by the weak law of large numbers:

• The probability that the sample mean deviates from the population mean by more than any positive amount tends towards zero as the sample size grows.
• This concept is crucial for statistical inference and allows us to estimate population parameters based on sample statistics.

Convergence of the sample mean ensures the reliability of average-based estimates in everyday data analysis.