Problem 7 - A Modern Introduction to Probability and Statistics: Understanding Why and How

Question

\(\square\) Suppose that \(x_{1}, x_{2}, \ldots, x_{n}\) is a dataset, which is a realization of a random sample from a Rayleigh distribution, which is a continuous distribution with probability density function given by $$ f_{\theta}(x)=\frac{x}{\theta^{2}} \mathrm{e}^{-\frac{1}{2} x^{2} / \theta^{2}} \quad \text { for } x \geq 0 . $$ In this case what is the maximum likelihood estimate for \(\theta\) ?

Step-by-Step Solution

Answer

The MLE for \(\theta\) is \(\sqrt{\frac{1}{2n} \sum_{i=1}^{n} x_i^2}\).

Step 1: Understanding the Problem

The problem requires finding the Maximum Likelihood Estimate (MLE) for the parameter \(\theta\) of the Rayleigh distribution, based on a given dataset \(x_{1}, x_{2}, \ldots, x_{n}\). The probability density function (PDF) of the Rayleigh distribution is provided as \(f_{\theta}(x) = \frac{x}{\theta^{2}} e^{-\frac{1}{2} x^{2} / \theta^{2}}\) for \(x \geq 0\).

Step 2: Write the Likelihood Function

The likelihood function \(L(\theta)\) for the sample \(x_{1}, x_{2}, \ldots, x_{n}\) is the product of the individual density functions: \[ L(\theta) = \prod_{i=1}^{n} \frac{x_i}{\theta^2} e^{-\frac{1}{2} \frac{x_i^2}{\theta^2}} \]. Simplifying this, we have \[ L(\theta) = \left(\frac{1}{\theta^2}\right)^n \prod_{i=1}^{n} x_i \cdot e^{-\frac{1}{2}\sum_{i=1}^{n} \frac{x_i^2}{\theta^2}} \].

Step 3: Take the Natural Logarithm

To simplify the process of finding the maximum likelihood estimate, take the natural logarithm of the likelihood function. This gives us the log-likelihood function: \[ \log L(\theta) = -2n \log \theta + \sum_{i=1}^{n} \log x_i - \frac{1}{2 \theta^2} \sum_{i=1}^{n} x_i^2 \].

Step 4: Differentiate the Log-Likelihood Function

Differentiate the log-likelihood function with respect to \(\theta\): \[ \frac{d}{d\theta} \log L(\theta) = -\frac{2n}{\theta} + \frac{1}{\theta^3} \sum_{i=1}^{n} x_i^2 \].

Step 5: Set the Derivative to Zero

Set the derivative equal to zero and solve for \(\theta\): \[ -\frac{2n}{\theta} + \frac{1}{\theta^3} \sum_{i=1}^{n} x_i^2 = 0 \]. Multiply through by \(\theta^3\) to get \( -2n\theta + \sum_{i=1}^{n} x_i^2 = 0 \), which simplifies to \(2n\theta = \sum_{i=1}^{n} x_i^2\).

Step 6: Solve for \(\theta\)

Rearrange the equation from the previous step to solve for \(\theta\): \[ \theta = \sqrt{\frac{1}{2n} \sum_{i=1}^{n} x_i^2} \]. This gives the maximum likelihood estimate for \(\theta\).

Key Concepts

Rayleigh Distribution

The Rayleigh distribution is a continuous probability distribution primarily used to model data in which the magnitude of a vector with two orthogonal components is the focus. It is particularly useful in signal processing, with applications in radar and communication systems.
For a random variable that follows a Rayleigh distribution, the probability density function (PDF) is given by:

• \( f_{\theta}(x) = \frac{x}{\theta^{2}} e^{-\frac{1}{2} x^{2} / \theta^{2}} \quad \text{for } x \geq 0 \)

This formula says that the likelihood of observing a specific value decreases exponentially as the value increases, with the parameter \(\theta\) shaping the distribution. Being able to identify and work with the Rayleigh distribution is valuable for understanding phenomena that are inherently two-dimensional, such as wind speed and its directional components, seismic wave magnitudes, or noise in acoustic environments.

Probability Density Function

The Probability Density Function (PDF) is a function that describes the likelihood of a random variable to take on a particular value. For continuous distributions, like the Rayleigh distribution, the PDF must satisfy two main properties:

• The area under the PDF curve from negative infinity to positive infinity equals 1, ensuring total probability is one.
  
• The value of the PDF is always non-negative, meaning \( f(x) \geq 0 \).
  

In the Rayleigh distribution, the PDF takes a specific form that depends on the parameter \(\theta\), which scales the distribution. We can relate the value of the PDF at any point \(x\) to the probability of observing values around \(x\). Although for continuous distributions, the probability at any exact point is zero, the PDF provides a relative likelihood that helps us understand and calculate probabilities over intervals.

Log-Likelihood Function

The log-likelihood function is crucial for Maximum Likelihood Estimation (MLE) as it simplifies the multiplication of terms into sums, making derivatives easier to compute. For a set of independent observations from a Rayleigh distribution, the likelihood function \(L(\theta)\) is the product of their individual PDFs. Taking the natural logarithm of \(L(\theta)\) gives:

\[\log L(\theta) = -2n \log \theta + \sum_{i=1}^{n} \log x_i - \frac{1}{2 \theta^2} \sum_{i=1}^{n} x_i^2\]The purpose is to express the likelihood in a form where differentiation is more straightforward. By using the log-likelihood, we can easily derive the equations needed to find the MLE of \(\theta\), as the summation and logarithm properties simplify differentiation and algebraic manipulation. Understanding how to convert the likelihood into its log version is an essential step for parameter estimation.

Differentiation

Differentiation is a mathematical technique used to compute the rate at which a function changes at any given point. In the context of the Maximum Likelihood Estimation, we differentiate the log-likelihood function with respect to \(\theta\) to find points that maximize the likelihood. This requires setting the derivative to zero, indicating potential maximums or minimums.

For the Rayleigh distribution, the differentiation of the log-likelihood function with respect to \(\theta\) yields:

• \[ \frac{d}{d\theta} \log L(\theta) = -\frac{2n}{\theta} + \frac{1}{\theta^3} \sum_{i=1}^{n} x_i^2 \]

By solving the resulting equation after setting it to zero, we arrive at a formula for \(\theta\) that is most likely to occur given our data. Successfully carrying out these calculations allows us to estimate parameters that best explain the observed data, an integral part of statistical analysis and inference.