Problem 6 - A Modern Introduction to Probability and Statistics: Understanding Why and How

Question

Let \(x_{1}, x_{2}, \ldots, x_{n}\) be a dataset that is a realization of a random sample from a distribution with probability density \(f_{\delta}(x)\) given by $$ f_{\delta}(x)= \begin{cases}\mathrm{e}^{-(x-\delta)} & \text { for } x \geq \delta \\ 0 & \text { for } x<\delta\end{cases} $$ a. Draw the likelihood \(L(\delta)\). b. Determine the maximum likelihood estimate for \(\delta\).

Step-by-Step Solution

Answer

The MLE of \( \delta \) is \( \hat{\delta} = \min(x_1, x_2, \ldots, x_n) \). The likelihood corresponds to an exponential curve starting at \( \hat{\delta} \).

Step 1: Understand the Probability Density Function

The probability density function (pdf) given, \( f_{\delta}(x) \), is defined piecewise. For \( x \geq \delta \), the function is \( \mathrm{e}^{-(x-\delta)} \). For \( x < \delta \), the pdf is zero. This implies that the distribution is an exponentially decaying function starting at \( x = \delta \).

Step 2: Write the Likelihood Function

The likelihood function \( L(\delta) \) for a dataset \( x_{1}, x_{2}, \ldots, x_{n} \) is the joint probability of observing the data given the parameter \( \delta \). Assuming independence, \( L(\delta) = \prod_{i=1}^{n} f_{\delta}(x_i) \). For \( x_i \geq \delta \), this becomes:\[ L(\delta) = \prod_{i=1}^{n} \mathrm{e}^{-(x_i-\delta)} = \mathrm{e}^{-\sum_{i=1}^{n}(x_i-\delta)}. \]Simplifying, the likelihood is:\[ L(\delta) = \mathrm{e}^{-\sum_{i=1}^{n} x_i + n\delta}. \]

Step 3: Simplify the Likelihood Function

The likelihood function can be further simplified by isolating terms:\[ L(\delta) = \mathrm{e}^{-S + n\delta}, \]where \( S = \sum_{i=1}^{n} x_i \). Since the exponential function is monotonic, maximizing \( L(\delta) \) is equivalent to maximizing the exponent:\( -S + n\delta \).

Step 4: Derive Maximum Likelihood Estimate

To find the maximum likelihood estimate (MLE) of \( \delta \), we need to maximize the expression \( n\delta - S \). Observing that the function is linear in \( \delta \), its maximum occurs at the largest possible value of \( \delta \) such that all \( x_i \geq \delta \). Thus, the MLE is:\[ \hat{\delta} = \min(x_1, x_2, \ldots, x_n). \]

Step 5: Conclusion

The steps above show that the MLE of \( \delta \) is the smallest observed data point \( x_i \) since increasing \( \delta \) decreases the exponent \( n\delta - S \) and because \( f_{\delta}(x) = 0 \) for any \( x_i < \delta \). Therefore, all data points must be greater than or equal to \( \hat{\delta} \).

Key Concepts

Likelihood Function

A likelihood function is a core concept in statistical estimation, especially in maximum likelihood estimation (MLE). It provides a way to find the parameter values that make the observed data most probable. When you have a likelihood function, you're assessing the plausibility of different parameter values given your data.

In our exercise, the likelihood function beyond all its math is intuitively asking: "How likely is it that we would observe our specific data, given the parameter \( \delta \)?" This means taking the joint probability of all the observed data points, given the parameter \( \delta \). Using the rule of independence, the likelihood function can be expressed as a product of probabilities for each data point.

To make calculations easier, it's common to work with the log of the likelihood function, which turns products into sums (this is known as the log-likelihood). In our example, the likelihood function \( L(\delta) = \mathrm{e}^{-\sum_{i=1}^{n} x_i + n\delta} \) is already in a form where the exponent can help find maxima or minima easily. This is important for identifying the maximum likelihood estimate, which is the core goal.

Probability Density Function

The probability density function (pdf) is a fundamental concept when dealing with continuous random variables. It tells us about the relative likelihood for a random variable to take on a given value.

In the context of our exercise, the pdf \( f_{\delta}(x) \) is given by:

• \( \mathrm{e}^{-(x-\delta)} \) for \( x \geq \delta \)
• 0 for \( x < \delta \)

This structure suggests that the distribution of our data has an exponentially decaying behavior that starts logging at \( x = \delta \).

The piecewise nature of the function implies that it's zero for any values less than \( \delta \), meaning those values are incompatible with the parameter \( \delta \). This significantly affects the shape of the likelihood function and, consequently, how we determine the maximum likelihood estimate. The pdf acts as a building block for constructing the likelihood function, directly impacting the estimation process.

Exponential Distribution

The exponential distribution is a probability distribution commonly used to model time until an event occurs, such as failure rates or arrival times. It's characterized by a constant hazard rate, meaning the rate of occurrence is constant over time.

In our exercise, the provided probability density function (pdf) \( f_{\delta}(x) = \mathrm{e}^{-(x-\delta)} \) for \( x \geq \delta \) is a form of an exponential distribution. This particular distribution starts at a location parameter \( \delta \).

• The exponential function \( \mathrm{e}^{-(x-\delta)} \) dictates that larger values of \( x \) (further away from \( \delta \)) have lower probabilities, showcasing the decaying nature of the distribution.
• The parameter \( \delta \) acts here as a shift or threshold, beyond which this decaying probability comes into play.

In maximum likelihood estimation contexts, fitting an exponential distribution means finding the parameter \( \delta \) where the observed data best aligns with this decay pattern. Thus, in our exercise, the smallest observed value \( x_i \) will be the maximum likelihood estimate, as it sets the point where the pdf starts recognizing observations.