MLE'nin ortalamanın önyargılı bir tahminini ürettiği bir örnek var mı?

17

Önyargılı olan ortalamanın MLE tahmincisine bir örnek verebilir misiniz?

MLE tahmincilerini genel olarak düzenlilik koşullarını ihlal ederek bozan bir örnek aramıyorum.

İnternette görebildiğim tüm örnekler varyansa işaret ediyor ve ortalama ile ilgili hiçbir şey bulamıyorum.

DÜZENLE

@MichaelHardy, belirli bir önerilen model altında MLE kullanarak tekdüze dağılım ortalamasının önyargılı bir tahminini aldığımız bir örnek verdi.

ancak

https://en.wikipedia.org/wiki/Uniform_distribution_(continuous)#Estimation_of_midpoint

MLE'nin, önerilen başka bir model altında, ortalamanın muntazam bir şekilde minimum tarafsız bir tahmincisi olduğunu ileri sürmektedir.

Bu noktada, model nötr olan örnek bir ortalama tahmin edicinin aksine, çok varsayılmış bir modele bağlıysa, MLE tahminiyle ne anlama geldiğini hala net değilim. Sonunda nüfus hakkında bir şey tahmin etmekle ilgileniyorum ve varsayılmış bir modelin parametresinin tahminini gerçekten umursamıyorum.

DÜZENLEME 2

@ChristophHanck modelin ek bilgi içeren önyargı gösterdiği gibi MSE'yi azaltmayı başaramadı.

Ayrıca ek sonuçlarımız var:

http://www.maths.manchester.ac.uk/~peterf/CSI_ch4_part1.pdf (p61) http://www.cs.tut.fi/~hehu/SSP/lecture6.pdf (slayt 2) http: / /www.stats.ox.ac.uk/~marchini/bs2a/lecture4_4up.pdf (slayt 5)

"Eğer efficient'nin en verimli bir tarafsız tahmincisi ˆθ varsa (yani unbiased tarafsız ve CRLB'ye eşit ise), o zaman maksimum tahmin yöntemi bunu üretecektir."

"Dahası, eğer etkili bir tahminci varsa, ML tahmincisidir."

Serbest model parametrelerine sahip MLE tarafsız ve verimli olduğu için, tanım gereği bu "" En Çok Olabilirlik Tahmincisi?

DÜZENLEME 3

@AlecosPapadopoulos'un matematik forumunda Half Normal dağılımına sahip bir örneği var.

/math/799954/can-the-maximum-likelihood-estimator-be-unbiased-and-fail-to-achieve-cramer-rao

Tek tip durumda olduğu gibi parametrelerinin hiçbirini tutturmaz. Ortalama kestiricinin yanlılığını göstermese de, bunun yerleştiğini söyleyebilirim.

maximum-likelihood mean bias

— Çağdaş Özgenç
kaynak

10

Sıfır ve teta üzerinde bir üniforma ortalaması.

— Christoph Hanck

1

"Nüfus hakkında bir şeyler tahmin etmek" ile "varsayılmış bir modelin parametresi" arasındaki farkınızı takip edemem. Tüm parametrik istatistiklerde, bir popülasyonu bazı parametrelerle parametrelendiririz. Tabii ki, bu yüzden yanlış tanımlama sorunları ile karşılaşabiliriz, ancak bu konu ile ilgili görünmüyor.

— Christoph Hanck

5

Örneğin, bir popülasyon, ortalama ve varyans (örneğin normal bir popülasyon için yeterli olacaktır) gibi parametreleri / momentleri ile karakterize edilebilir. Ve: İnsanların sizinle bu forumdaki herkesten daha fazla veya daha az bilgiç olduğunu düşünmüyorum.

— Christoph Hanck

2

Eğer "parametre" ve "ortalama" arasında geçişin belirgin el çabukluğu konusunda mutsuz hissediyorsanız , yoğunluk

ile ortalama

cinsinden belirli bir negatif olmayan dağılımı tanımlayayım.

μ

$\mu$

\frac{1}{2 μ}

$\frac{1}{2\mu}$ verdiği destek üzerine

[0, 2 μ]

$[0, 2\mu]$ ...

— Gümüş Balık

1

Düzenlemeniz 2 ile ilgili olarak, bu sonuçların birçoğu, örnek iş parçasının parametreye bağlı olduğu bu iş parçacığında tartışılan tek tip örnek için tatmin olmayan düzenlilik koşulları altında türetilir.

— Christoph Hanck

32

Christoph Hanck, önerdiği örneğin ayrıntılarını yayınlamadı. I o aralık üniform dağılım anlamına gelir almak $[0,\theta],$ bir iid örnek göre $X_1,\ldots,X_n$ boyutunun daha fazla $n=1.$

Ortalama $\theta/2$ .

Ortalamanın MLE değeri $\max\{X_1,\ldots,X_n\}/2.$

olduğu için önyargılıdır $\Pr(\max < \theta) = 1,$ dolayısıyla $\operatorname{E}({\max}/2)<\theta/2.$

Not: Belki de ortalama en iyi yansız tahmincisinin örnek ortalama olmadığını , $\theta/2$ olduğunu belirtmeliyiz .Örnek ortalaması bir kötü bir tahmin olupBazı numuneler için, örnek ortalaması daha az olduğu için

\frac{n + 1}{2 n} \cdot max {X_{1}, \dots, X_{n}} .

$\frac{n+1} {2n} \cdot \max\{X_1,\ldots,X_n\}.$

θ / 2

$\theta/2$

ve açık bir şekilde mümkün değildir

daha az olması

PS sonunda

\frac{1}{2} max {X_{1}, \dots, X_{n}},

$\dfrac 1 2 \max\{X_1,\ldots,X_n\},$

θ / 2

$\theta/2$

max / 2.

${\max}/2.$

Pareto dağılımının böyle bir durum olduğundan şüpheleniyorum. Olasılık ölçüsü: Beklenen değer

α {(\frac{κ}{x})}^{α} \frac{d x}{x} for x > κ .

$\alpha\left( \frac \kappa x \right)^\alpha\ \frac{dx} x \text{ for } x >\kappa.$

Beklenen değerin MLE değeri

\frac{α}{α - 1} κ .

$\dfrac \alpha {\alpha -1 } \kappa.$

burada

\frac{n}{n - \sum_{i = 1}^{n} ((\log X_{i}) - \log (min))} \cdot min

$\frac n {n - \sum_{i=1}^n \big((\log X_i) - \log(\min)\big)} \cdot \min$

min = min {X_{1}, \dots, X_{n}} .

$\min = \min\{X_1,\ldots,X_n\}.$

I haven't worked out the expected value of the MLE for the mean, so I don't know what its bias is.

— Michael Hardy
kaynak

12

Cagdas, It's not legitimate to ask for a countexample and then deny that you would propose something else! It's like asking for an example of a fruit that is not red, being shown a blueberry, and then saying it doesn't count because you don't like blueberries.

— whuber

7

That's not relevant to the question you asked.

— whuber

8

@CagdasOzgenc : Whether the MLE is biased or not depends on the model. There's no such thing as an MLE without a model. And if you alter the model, you alter the MLE.

— Michael Hardy

8

@CagdasOzgenc Here's a socratic question: the sample mean is an unbiased estimator of what? You need a model to have a parameter to be estimating.

— Matthew Drury

9

The mean of an i.i.d. sample is an unbiased estimator of the population mean, but one cannot speak of a maximum-likelihood estimator of anything without more structure than what is needed to speak of an unbiased estimator of something.

— Michael Hardy

18

Here's an example that I think some may find surprising:

In logistic regression, for any finite sample size with non-deterministic outcomes (i.e. $0 < p_{i} < 1$ ), any estimated regression coefficient is not only biased, the mean of the regression coefficient is actually undefined.

This is because for any finite sample size, there is a positive probability (albeit very small if the number of samples is large compared with the number of regression parameters) of getting perfect separation of outcomes. When this happens, estimated regression coefficients will be either $-\infty$ or $\infty$ . Having positive probability of being either $-\infty$ or $\infty$ implies the expected value is undefined.

For more on this particular issue, see the Hauck-Donner-effect.

— Cliff AB
kaynak

1

This is quite clever. I wonder if the MLE of logistic regression coefficients is unbiased conditional on the nonoccurence of the Hauck-Donner effect?

— gung - Reinstate Monica

3

@gung: Short answer: ignoring the Hauck-Donner effect, there is still upward bias in absolute regression coefficients (i.e. negative coefficients have downward bias, positive have upward bias). Interestingly, there appears to be a bias toward 0.5 in estimated probabilities. I've started writing up about it on this post, but haven't put up my results on the biases of the estimated probabilities.

— Cliff AB

10

Although @MichaelHardy has made the point, here is a more detailed argument as to why the MLE of the maximum (and hence, that of the mean $\theta/2$ , by invariance) is not unbiased, although it is in a different model (see the edit below).

We estimate the upper bound of the uniform distribution $U[0,\theta]$ . Here, $y_{(n)}$ is the MLE, for a random sample $y$ . We show that $y_{(n)}$ is not unbiased. Its cdf is

\begin{array}{rcl} F_{y_{(n)}} (x) & = & Pr {Y_{1} ⩽ x, \dots, Y_{n} ⩽ x} \\ = & Pr {Y_{1} ⩽ x}^{n} \\ = & {\begin{cases} 0 & for x < 0 \\ {(\frac{x}{θ})}^{n} & for 0 ⩽ x ⩽ θ \\ 1 & for x > θ \end{cases} \end{array}

$\begin{eqnarray*} F_{y_{(n)}}(x)&=&\Pr\{Y_1\leqslant x,\ldots,Y_n\leqslant x\}\\ &=&\Pr\{Y_1\leqslant x\}^n\\ &=&\begin{cases} 0&\qquad\text{for}\quad x<0\\ \left(\frac{x}{\theta}\right)^n&\qquad\text{for}\quad 0\leqslant x\leqslant\theta\\ 1&\qquad\text{for}\quad x>\theta \end{cases} \end{eqnarray*}$ Thus, its density is

f_{y_{(n)}} (x) = {\begin{cases} \frac{n}{θ} {(\frac{x}{θ})}^{n - 1} & for 0 ⩽ x ⩽ θ \\ 0 & else \end{cases}

$f_{y_{(n)}}(x)= \begin{cases} \frac{n}{\theta}\left(\frac{x}{\theta}\right)^{n-1}&\qquad\text{for}\quad 0\leqslant x\leqslant\theta\\ 0&\qquad\text{else} \end{cases}$ Hence,

\begin{array}{rcl} E [Y_{(n)}] & = & \int_{0}^{θ} x \frac{n}{θ} {(\frac{x}{θ})}^{n - 1} d x \\ = & \int_{0}^{θ} n {(\frac{x}{θ})}^{n} d x \\ = & \frac{n}{n + 1} θ \end{array}

$\begin{eqnarray*} E[Y_{(n)}]&=&\int_0^\theta x\frac{n}{\theta}\left(\frac{x}{\theta}\right)^{n-1}dx\\ &=&\int_0^\theta n\left(\frac{x}{\theta}\right)^{n}dx\\ &=&\frac{n}{n+1}\theta \end{eqnarray*}$

EDIT: It is indeed the case that (see the discussion in the comments) the MLE is unbiased for the mean in the case in which both the lower bound $a$ and upper bound $b$ are unknown. Then, the minimum $Y_{(1)}$ is the MLE for $a$ , with (details omitted) expected value

E (Y_{(1)}) = \frac{n a + b}{n + 1}

$E(Y_{(1)})=\frac{na+b}{n+1}$ while

E (Y_{(n)}) = \frac{n b + a}{n + 1}

$E(Y_{(n)})=\frac{nb+a}{n+1}$ so that the MLE for

(a + b) / 2

$(a+b)/2$ is

\frac{Y_{(1)} + Y_{(n)}}{2}

$\frac{Y_{(1)}+Y_{(n)}}{2}$ with expected value

E (\frac{Y_{(1)} + Y_{(n)}}{2}) = \frac{n a + b + n b + a}{2 (n + 1)} = \frac{a + b}{2}

$E\left(\frac{Y_{(1)}+Y_{(n)}}{2}\right)=\frac{na+b+nb+a}{2(n+1)}=\frac{a+b}{2}$

EDIT 2: To elaborate on Henry's point, here is a little simulation for the MSE of the estimators of the mean, showing that while the MLE if we do not know the lower bound is zero is unbiased, the MSEs for the two variants are identical, suggesting that the estimator which incorporates knowledge of the lower bound reduces variability.

theta <- 1
mean <- theta/2
reps <- 500000
n <- 5
mse <- bias <- matrix(NA, nrow = reps, ncol = 2)

for (i in 1:reps){
  x <- runif(n, min = 0, max = theta)
  mle.knownlowerbound <- max(x)/2
  mle.unknownlowerbound <- (max(x)+min(x))/2
  mse[i,1] <- (mle.knownlowerbound-mean)^2
  mse[i,2] <- (mle.unknownlowerbound-mean)^2
  bias[i,1] <- mle.knownlowerbound-mean
  bias[i,2] <- mle.unknownlowerbound-mean

}

> colMeans(mse)
[1] 0.01194837 0.01194413

> colMeans(bias)
[1] -0.083464968 -0.000121968

— Christoph Hanck
kaynak

Because Wikipedia is proposing a different model to begin with. That's where my confusion lies.

— Cagdas Ozgenc

Yes, but once we adjust to the special case discussed here, namely

a = 0

$a=0$ , we are back at square 1. In that case, we do not need the sample minimum for estimation anymore, as we know that the lower bound is zero, so that the MLE of the midpoint (=median=mean) simply becomes

(m a x + 0) / 2

$(max+0)/2$ again.

— Christoph Hanck

2

I have not worked out the details, but the MLE in that model could be unbiased if the minimum overestimates the lower bound by the same amount as the maximum underestimates the maximum, so that the midpoint is being estimated without bias.

— Christoph Hanck

4

@CagdasOzgenc: unbiasedness is not the only or even the most important measure of better. By knowing one end of the support precisely, you may lose the balance between errors in estimating the mean, but you end up with (for example) a better estimate of the range

— Henry

6

Maximum likelihood estimators are not always "best" across all criteria for small sample sizes. So what? They don't pretend to be, either. If you want to use a different estimator for your problem that has better properties according to some criterion for sample sizes that are in the neighborhood of your actual sample size, you're free to do so. I do so, and so do other people. No one is claiming that using MLE is justified in all situations just because it's MLE.

— jbowman

5

Completing here the omission in my answer over at math.se referenced by the OP,

assume that we have an i.i.d. sample of size $n$ of random variables following the Half Normal distribution. The density and moments of this distribution are

f_{H} (x) = \sqrt{2 / π} \cdot \frac{1}{v^{1 / 2}} \cdot \exp {- \frac{x^{2}}{2 v}} E (X) = \sqrt{2 / π} \cdot v^{1 / 2} \equiv μ, Var (X) = (1 - \frac{2}{π}) v

$f_H(x) = \sqrt{2/\pi}\cdot \frac 1{v^{1/2}}\cdot \exp\big\{-\frac {x^2}{2v} \big\} \\ E(X) = \sqrt{2/\pi}\cdot v^{1/2}\equiv \mu,\;\; \operatorname{Var}(X) = \left(1-\frac 2 \pi \right)v$

The log-likelihood of the sample is

L (v ∣ x) = n \ln \sqrt{2 / π} - \frac{n}{2} \ln v - \frac{1}{2 v} \sum_{i = 1}^{n} x_{i}^{2}

$L(v\mid \mathbf x) = n\ln\sqrt{2/\pi}-\frac n2\ln v -\frac 1 {2v} \sum_{i=1}^n x_i^2$

The first derivative with respect to $v$ is

\frac{\partial}{\partial v} L (v ∣ x) = - \frac{n}{2 v} + \frac{1}{2 v^{2}} \sum_{i = 1}^{n} x_{i}^{2}, ⟹ {\hat{v}}_{MLE} = \frac{1}{n} \sum_{i = 1}^{n} x_{i}^{2}

$\frac {\partial}{\partial v}L(v\mid\mathbf x) = -\frac n{2v} + \frac 1 {2v^2} \sum_{i=1}^n x_i^2,\implies \hat v_\text{MLE} = \frac 1n \sum_{i=1}^nx_i^2$

so it is a method of moments estimator. It is unbiased since,

E ({\hat{v}}_{MLE}) = E (X^{2}) = Var (X) + [E (X)])^{2} = (1 - \frac{2}{π}) v + \frac{2}{π} v = v

$E(\hat v_\text{MLE}) = E(X^2) = \operatorname{Var}(X) + [E(X)])^2 = \left(1-\frac 2 \pi \right)v + \frac 2 \pi v = v$

But, the resulting estimator for the mean is downward biased due to Jensen's inequality

\begin{aligned} {\hat{μ}}_{MLE} = \sqrt{2 / π} \cdot \sqrt{{\hat{v}}_{MLE}} ⟹ & E ({\hat{μ}}_{MLE}) = \sqrt{2 / π} \cdot E (\sqrt{{\hat{v}}_{MLE}}) \\ < \sqrt{2 / π} \cdot [\sqrt{E ({\hat{v}}_{MLE})}] = \sqrt{2 / π} \cdot \sqrt{v} = μ \end{aligned}

$\begin{align} \hat \mu_\text{MLE} = \sqrt{2/\pi}\cdot \sqrt {\hat v_\text{MLE}} \implies & E\left(\hat \mu_\text{MLE}\right) = \sqrt{2/\pi}\cdot E\left(\sqrt {\hat v_\text{MLE}}\,\right) \\[6pt] & < \sqrt{2/\pi}\cdot \left[\sqrt {E(\hat v_\text{MLE})}\,\right] = \sqrt{2/\pi}\cdot \sqrt v = \mu \end{align}$

— Alecos Papadopoulos
kaynak

4

The famous Neyman Scott problem has an inconsistent MLE in that it never even converges to the right thing. Motivates the use of conditional likelihood.

Take $(X_i, Y_i) \sim \mathcal{N}\left(\mu_i, \sigma^2 \right)$ . The MLE of $\mu_i$ is $(X_i + Y_i)/2$ and of $\sigma^2$ is $\hat{\sigma}^2 = \sum_{i=1}^n \frac{1}{n} s_i^2$ with $s_i^2 = (X_i - \hat{\mu}_i)^2/2 + (Y_i - \hat{\mu}_i)^2/2 = (X_i - Y_i)^2 / 4$ which has expected value $\sigma^2/4$ and so biased by a factor of 2.

— AdamO
kaynak

2

While this example holds true, this actually defies one of the basic regularity conditions for asymptotic results of MLE's: that

k / n \to 0

$k / n \rightarrow 0$ , where

k

$k$ is the number of parameters estimated and

n

$n$ is the sample size.

— Cliff AB

1

@CliffAB the assumption violation is that the parametric dimension is not fixed. The dimension of

Θ

$\Theta$ goes to

\infty

$\infty$ as

n \to \infty

$n \rightarrow \infty$ . I think that's what you're saying, but don't know what

k

$k$ means. The practical illustration of this example of course is that these results would be biased even in small samples and you have to use conditional likelihood, like a mixed effects model, to estimate

σ

$\sigma$ in this case.

— AdamO

3

There is an infinite range of examples for this phenomenon since

the maximum likelihood estimator of a bijective transform $\Psi(\theta)$ of a parameter $\theta$ is the bijective transform of the maximum likelihood estimator of $\theta$ , $\Psi(\hat{\theta}_\text{MLE})$ ;
the expectation of the bijective transform of the maximum likelihood estimator of $\theta$ , $\Psi(\hat{\theta}_\text{MLE})$ , $\mathbb{E}[\Psi(\hat{\theta}_\text{MLE})]$ is not the bijective transform of the expectation of the maximum likelihood estimator, $\Psi(\mathbb{E}[\hat{\theta}_\text{MLE}])$ ;
most transforms $\Psi(\theta)$ are expectations of some transform of the data, $\mathfrak{h}(X)$ , at least for exponential families, provided an inverse Laplace transform can be applied to them.

— Xi'an
kaynak