koşullu OLS tahmincisinin

Bunu biliyorum

β 0^= y ¯ - β 1^x ¯

$\hat{\beta_0}=\bar{y}-\hat{\beta_1}\bar{x}$ ve bu ı varyans hesaplanan zaman var ne kadar geçerli:

V a r (β 0^) = V a r (y ¯ - β 1^x ¯) = V a r ((- x ¯) β 1^+ y ¯) = V a r ((- x ¯) β 1^) + V a r (y ¯) = (- x ¯) 2 V a r (β 1^) + 0 = (x ¯) 2 V a r (β 1^) + 0 = σ 2 ( x ¯ ) 2 \sum i = 1 n ( x i - x ¯ ) 2

$\begin{align*} Var(\hat{\beta_0}) &= Var(\bar{y} - \hat{\beta_1}\bar{x}) \\ &= Var((-\bar{x})\hat{\beta_1}+\bar{y}) \\ &= Var((-\bar{x})\hat{\beta_1})+Var(\bar{y}) \\ &= (-\bar{x})^2 Var(\hat{\beta_1}) + 0 \\ &= (\bar{x})^2 Var(\hat{\beta_1}) + 0 \\ &= \frac{\sigma^2 (\bar{x})^2}{\displaystyle\sum\limits_{i=1}^n (x_i - \bar{x})^2} \end{align*}$

ama bu benim sahip olduğum kadar uzak. Hesaplamaya çalıştığım son formül

V a r (β 0^) = σ 2 n - 1 \sum i = 1 n x 2 i \sum i = 1 n ( x i - x ¯ ) 2

$\begin{align*} Var(\hat{\beta_0}) &= \frac{\sigma^2 n^{-1}\displaystyle\sum\limits_{i=1}^n x_i^2}{\displaystyle\sum\limits_{i=1}^n (x_i - \bar{x})^2} \end{align*}$

Nasıl alınacağından emin değilim

(x ¯) 2 = 1 n \sum i = 1 n x 2 i

$(\bar{x})^2 = \frac{1}{n}\displaystyle\sum\limits_{i=1}^n x_i^2$ matematik orada doğru yukarı olduğunu varsayarak.

Bu doğru yol mu?

(x ¯) 2 = (1 n \sum i = 1 n x i) 2 = 1 n 2 (\sum i = 1 n x i) 2

$\begin{align} (\bar{x})^2 &= \left(\frac{1}{n}\displaystyle\sum\limits_{i=1}^n x_i\right)^2 \\ &= \frac{1}{n^2} \left(\displaystyle\sum\limits_{i=1}^n x_i\right)^2 \end{align}$

I'm sure it's simple, so the answer can wait for a bit if someone has a hint to push me in the right direction.

regression self-study

— M T
kaynak

This is not the right path. The 4th equation doesn't hold. For example, with

x1=−1 $x_1=−1$ ,

x2=0 $x_2=0$ , and

x3=1 $x_3=1$ , the left term is zero, whilst the right term is

2/3 $2/3$ . The problem comes from the step where you split the variance (3rd line of second equation). See why?

— QuantIbex

Hint towards Quantlbex point: variance is not a linear function. It violates both additivity and scalar multiplication.

— David Marx

@DavidMarx That step should be

= V a r ((- x ¯) β 1^+ y ¯) = (x ¯) 2 V a r (β 1^) + y ¯

$=Var((-\bar{x})\hat{\beta_1}+\bar{y})=(\bar{x})^2Var(\hat{\beta_1})+\bar{y}$ , I think, and then once I substitute in for

β1^ $\hat{\beta_1}$ and

y¯ $\bar{y}$ (not sure what to do for this but I'll think about it more), that should put me on the right path I hope.

— M T

This is not correct. Think about the condition required for the variance of a sum to be equal to the sum of the variances.

— QuantIbex

No,

y¯ $\bar{y}$ is random since

yi=β0+β1xi+ϵ $y_i = \beta_0 + \beta_1 x_i + \epsilon$ , where

ϵ $\epsilon$ denotes the (random) noise. But OK, my previous comment was maybe misleading. Also,

Var(aX+b)=a2Var(X) ${\rm Var}(aX + b)= a^2{\rm Var}(X)$ , if

a $a$ and

b $b$ denote constants.

— QuantIbex

Yanıtlar:

This is a self-study question, so I provide hints that will hopefully help to find the solution, and I'll edit the answer based on your feedbacks/progress.

The parameter estimates that minimize the sum of squares are

β^0 β^1 = y ¯ - β^1 x ¯, = \sum n i = 1 ( x i - x ¯ ) y i \sum n i = 1 ( x i - x ¯ ) 2 .

$\begin{align} \hat{\beta}_0 &= \bar{y} - \hat{\beta}_1 \bar{x} , \\ \hat{\beta}_1 &= \frac{ \sum_{i = 1}^n(x_i - \bar{x})y_i }{ \sum_{i = 1}^n(x_i - \bar{x})^2 } . \end{align}$ To get the variance of

β^0 $\hat{\beta}_0$ , start from its expression and substitute the expression of

β^1 $\hat{\beta}_1$ , and do the algebra

V a r (β^0) = V a r (Y ¯ - β^1 x ¯) = \dots

${\rm Var}(\hat{\beta}_0) = {\rm Var} (\bar{Y} - \hat{\beta}_1 \bar{x}) = \ldots$

Edit:
We have

V a r (β^0) = V a r (Y ¯ - β^1 x ¯) = V a r (Y ¯) + (x ¯) 2 V a r (β^1) - 2 x ¯ C o v (Y ¯, β^1) .

$\begin{align} {\rm Var}(\hat{\beta}_0) &= {\rm Var} (\bar{Y} - \hat{\beta}_1 \bar{x}) \\ &= {\rm Var} (\bar{Y}) + (\bar{x})^2 {\rm Var} (\hat{\beta}_1) - 2 \bar{x} {\rm Cov} (\bar{Y}, \hat{\beta}_1). \end{align}$ The two variance terms are

V a r (Y ¯) = V a r (1 n \sum i = 1 n Y i) = 1 n 2 \sum i = 1 n V a r (Y i) = σ 2 n,

${\rm Var} (\bar{Y}) = {\rm Var} \left(\frac{1}{n} \sum_{i = 1}^n Y_i \right) = \frac{1}{n^2} \sum_{i = 1}^n {\rm Var} (Y_i) = \frac{\sigma^2}{n},$ and

V a r (β^1) = 1 [ \sum n i = 1 ( x i - x ¯ ) 2 ] 2 \sum i = 1 n (x i - x ¯) 2 V a r (Y i) = σ 2 \sum n i = 1 ( x i - x ¯ ) 2,

$\begin{align} {\rm Var} (\hat{\beta}_1) &= \frac{ 1 }{ \left[\sum_{i = 1}^n(x_i - \bar{x})^2 \right]^2 } \sum_{i = 1}^n(x_i - \bar{x})^2 {\rm Var} (Y_i) \\ &= \frac{ \sigma^2 }{ \sum_{i = 1}^n(x_i - \bar{x})^2 } , \end{align}$ and the covariance term is

Cov(Y¯,β^1)=Cov{1n∑i=1nYi,∑nj=1(xj−x¯)Yj∑ni=1(xi−x¯)2}=1n1∑ni=1(xi−x¯)2Cov{∑i=1nYi,∑j=1n(xj−x¯)Yj}=1n∑ni=1(xi−x¯)2∑i=1n(xj−x¯)∑j=1nCov(Yi,Yj)=1n∑ni=1(xi−x¯)2∑i=1n(xj−x¯)σ2=0

$\begin{align} {\rm Cov} (\bar{Y}, \hat{\beta}_1) &= {\rm Cov} \left\{ \frac{1}{n} \sum_{i = 1}^n Y_i, \frac{ \sum_{j = 1}^n(x_j - \bar{x})Y_j }{ \sum_{i = 1}^n(x_i - \bar{x})^2 } \right \} \\ &= \frac{1}{n} \frac{ 1 }{ \sum_{i = 1}^n(x_i - \bar{x})^2 } {\rm Cov} \left\{ \sum_{i = 1}^n Y_i, \sum_{j = 1}^n(x_j - \bar{x})Y_j \right\} \\ &= \frac{ 1 }{ n \sum_{i = 1}^n(x_i - \bar{x})^2 } \sum_{i = 1}^n (x_j - \bar{x}) \sum_{j = 1}^n {\rm Cov}(Y_i, Y_j) \\ &= \frac{ 1 }{ n \sum_{i = 1}^n(x_i - \bar{x})^2 } \sum_{i = 1}^n (x_j - \bar{x}) \sigma^2 \\ &= 0 \end{align}$ since

∑ni=1(xj−x¯)=0 $\sum_{i = 1}^n (x_j - \bar{x})=0$ .
And since

\sum i = 1 n (x i - x ¯) 2 = \sum i = 1 n x 2 i - 2 x ¯ \sum i = 1 n x i + \sum i = 1 n x ¯ 2 = \sum i = 1 n x 2 i - n x ¯ 2,

$\sum_{i = 1}^n(x_i - \bar{x})^2 = \sum_{i = 1}^n x_i^2 - 2 \bar{x} \sum_{i = 1}^n x_i + \sum_{i = 1}^n \bar{x}^2 = \sum_{i = 1}^n x_i^2 - n \bar{x}^2,$ we have

Var(β^0)=σ2n+σ2x¯2∑ni=1(xi−x¯)2=σ2n∑ni=1(xi−x¯)2{∑i=1n(xi−x¯)2+nx¯2}=σ2∑ni=1x2in∑ni=1(xi−x¯)2.

$\begin{align} {\rm Var}(\hat{\beta}_0) &= \frac{\sigma^2}{n} + \frac{ \sigma^2 \bar{x}^2}{ \sum_{i = 1}^n(x_i - \bar{x})^2 } \\ &= \frac{\sigma^2 }{ n \sum_{i = 1}^n(x_i - \bar{x})^2 } \left\{ \sum_{i = 1}^n(x_i - \bar{x})^2 + n \bar{x}^2 \right\} \\ &= \frac{\sigma^2 \sum_{i = 1}^n x_i^2}{ n \sum_{i = 1}^n(x_i - \bar{x})^2 }. \end{align}$

Edit 2

Why do we have ${\rm var} ( \sum_{i = 1}^n Y_i) = \sum_{i = 1}^n {\rm Var} (Y_i)$ ?

The assumed model is $Y_i = \beta_0 + \beta_1 X_i + \epsilon_i$ , where the $\epsilon_i$ are independant and identically distributed random variables with ${\rm E}(\epsilon_i) = 0$ and ${\rm var}(\epsilon_i) = \sigma^2$ .

Once we have a sample, the $X_i$ are known, the only random terms are the $\epsilon_i$ . Recalling that for a random variable $Z$ and a constant $a$ , we have ${\rm var}(a+Z) = {\rm var}(Z)$ . Thus,

v a r (\sum i = 1 n Y i) = v a r (\sum i = 1 n β 0 + β 1 X i + ϵ i) = v a r (\sum i = 1 n ϵ i) = \sum i = 1 n \sum j = 1 n c o v (ϵ i, ϵ j) = \sum i = 1 n c o v (ϵ i, ϵ i) = \sum i = 1 n v a r (ϵ i) = \sum i = 1 n v a r (β 0 + β 1 X i + ϵ i) = \sum i = 1 n v a r (Y i) .

$\begin{align} {\rm var} \left( \sum_{i = 1}^n Y_i \right) &= {\rm var} \left( \sum_{i = 1}^n \beta_0 + \beta_1 X_i + \epsilon_i \right)\\ &= {\rm var} \left( \sum_{i = 1}^n \epsilon_i \right) = \sum_{i = 1}^n \sum_{j = 1}^n {\rm cov} (\epsilon_i, \epsilon_j)\\ &= \sum_{i = 1}^n {\rm cov} (\epsilon_i, \epsilon_i) = \sum_{i = 1}^n {\rm var} (\epsilon_i)\\ &= \sum_{i = 1}^n {\rm var} (\beta_0 + \beta_1 X_i + \epsilon_i) = \sum_{i = 1}^n {\rm var} (Y_i).\\ \end{align}$ The 4th equality holds as

cov(ϵi,ϵj)=0 ${\rm cov} (\epsilon_i, \epsilon_j) = 0$ for

i≠j $i \neq j$ by the independence of the

ϵi $\epsilon_i$ .

— QuantIbex
kaynak

I think I got it! The book has suggested steps, and I was able to prove each step separately (I think). It's not as satisfying as just sitting down and grinding it out from this step, since I had to prove intermediate conclusions for it to help, but I think everything looks good.

— M T

See edit for the development of the suggested approach.

— QuantIbex

The variance of the sum equals the sum of the variances in this step:

V a r (Y ¯) = V a r (1 n \sum i = 1 n Y i) = 1 n 2 \sum i = 1 n V a r (Y i)

${\rm Var} (\bar{Y}) = {\rm Var} \left(\frac{1}{n} \sum_{i = 1}^n Y_i \right) = \frac{1}{n^2} \sum_{i = 1}^n {\rm Var} (Y_i)$ because since the

Xi $X_i$ are independent, this implies that the

Yi $Y_i$ are independent as well, right?

— M T

Also, you can factor out a constant from the covariance in this step:

1 n 1 \sum n i = 1 ( x i - x ¯ ) 2 C o v {\sum i = 1 n Y i, \sum j = 1 n (x j - x ¯) Y j}

$\frac{1}{n} \frac{ 1 }{ \sum_{i = 1}^n(x_i - \bar{x})^2 } {\rm Cov} \left\{ \sum_{i = 1}^n Y_i, \sum_{j = 1}^n(x_j - \bar{x})Y_j \right\}$ even though it's not in both elements because the formula for covariance is multiplicative, right?

— M T

@oort, in the numerator you have the sum of

n $n$ terms that are identical (and equal to

σ2 $\sigma^2$ ), so the numerator is

nσ2 $n \sigma^2$ .

— QuantIbex

I got it! Well, with help. I found the part of the book that gives steps to work through when proving the $Var \left( \hat{\beta}_0 \right)$ formula (thankfully it doesn't actually work them out, otherwise I'd be tempted to not actually do the proof). I proved each separate step, and I think it worked.

I'm using the book's notation, which is:

S S T x = \sum i = 1 n (x i - x ¯) 2,

$SST_x = \displaystyle\sum\limits_{i=1}^n (x_i - \bar{x})^2,$ and

ui $u_i$ is the error term.

1) Show that $\hat{\beta}_1$ can be written as $\hat{\beta}_1 = \beta_1 + \displaystyle\sum\limits_{i=1}^n w_i u_i$ where $w_i = \frac{d_i}{SST_x}$ and $d_i = x_i - \bar{x}$ .

This was easy because we know that

β^1 = β 1 + \sum i = 1 n ( x i - x ¯ ) u i S S T x = β 1 + \sum i = 1 n d i S S T x u i = β 1 + \sum i = 1 n w i u i

$\begin{align} \hat{\beta}_1 &= \beta_1 + \frac{\displaystyle\sum\limits_{i=1}^n (x_i - \bar{x}) u_i}{SST_x} \\ &= \beta_1 + \displaystyle\sum\limits_{i=1}^n \frac{d_i}{SST_x} u_i \\ &= \beta_1 + \displaystyle\sum\limits_{i=1}^n w_i u_i \end{align}$

2) Use part 1, along with $\displaystyle\sum\limits_{i=1}^n w_i = 0$ to show that $\hat{\beta_1}$ and $\bar{u}$ are uncorrelated, i.e. show that $E[(\hat{\beta_1}-\beta_1) \bar{u}] = 0$ .

E [(β 1^- β 1) u ¯] = E [u ¯ \sum i = 1 n w i u i] = \sum i = 1 n E [w i u ¯ u i] = \sum i = 1 n w i E [u ¯ u i] = 1 n \sum i = 1 n w i E (u i \sum j = 1 n u j) = 1 n \sum i = 1 n w i [E (u i u 1) + \dots + E (u i u j) + \dots + E (u i u n)]

$\begin{align} E[(\hat{\beta_1}-\beta_1) \bar{u}] &= E[\bar{u}\displaystyle\sum\limits_{i=1}^n w_i u_i] \\ &=\displaystyle\sum\limits_{i=1}^n E[w_i \bar{u} u_i] \\ &=\displaystyle\sum\limits_{i=1}^n w_i E[\bar{u} u_i] \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i E\left(u_i\displaystyle\sum\limits_{j=1}^n u_j\right) \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i \left[E\left(u_i u_1\right) +\cdots + E(u_i u_j) + \cdots+ E\left(u_i u_n \right)\right] \\ \end{align}$

and because the $u$ are i.i.d., $E(u_i u_j) = E(u_i) E(u_j)$ when $j \neq i$ .

When $j = i$ , $E(u_i u_j) = E(u_i^2)$ , so we have:

= 1 n \sum i = 1 n w i [E (u i) E (u 1) + \dots + E (u 2 i) + \dots + E (u i) E (u n)] = 1 n \sum i = 1 n w i E (u 2 i) = 1 n \sum i = 1 n w i [V a r (u i) + E (u i) E (u i)] = 1 n \sum i = 1 n w i σ 2 = σ 2 n \sum i = 1 n w i = σ 2 n \cdot S S T x \sum i = 1 n (x i - x ¯) = σ 2 n \cdot S S T x (0) = 0

$\begin{align} &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i \left[E(u_i) E(u_1) +\cdots + E(u_i^2) + \cdots + E(u_i) E(u_n)\right] \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i E(u_i^2) \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i \left[Var(u_i) + E(u_i) E(u_i)\right] \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i \sigma^2 \\ &= \frac{\sigma^2}{n}\displaystyle\sum\limits_{i=1}^n w_i \\ &= \frac{\sigma^2}{n \cdot SST_x}\displaystyle\sum\limits_{i=1}^n (x_i - \bar{x}) \\ &= \frac{\sigma^2}{n \cdot SST_x} \left(0\right) &= 0 \end{align}$

3) Show that $\hat{\beta_0}$ can be written as $\hat{\beta_0} = \beta_0 + \bar{u} - \bar{x}(\hat{\beta_1} - \beta_1)$ . This seemed pretty easy too:

β 0^= y ¯ - β 1^x ¯ = (β 0 + β 1 x ¯ + u ¯) - β 1^x ¯ = β 0 + u ¯ - x ¯ (β 1^- β 1) .

$\begin{align} \hat{\beta_0} &= \bar{y} - \hat{\beta_1} \bar{x} \\ &= (\beta_0 + \beta_1 \bar{x} + \bar{u}) - \hat{\beta_1} \bar{x} \\ &= \beta_0 + \bar{u} - \bar{x}(\hat{\beta_1} - \beta_1). \end{align}$

4) Use parts 2 and 3 to show that $Var(\hat{\beta_0}) = \frac{\sigma^2}{n} + \frac{\sigma^2 (\bar{x}) ^2} {SST_x}$ :

V a r (β 0^) = V a r (β 0 + u ¯ - x ¯ (β 1^- β 1)) = V a r (u ¯) + (- x ¯) 2 V a r (β 1^- β 1) = σ 2 n + (x ¯) 2 V a r (β 1^) = σ 2 n + σ 2 ( x ¯ ) 2 S S T x .

$\begin{align} Var(\hat{\beta_0}) &= Var(\beta_0 + \bar{u} - \bar{x}(\hat{\beta_1} - \beta_1)) \\ &= Var(\bar{u}) + (-\bar{x})^2 Var(\hat{\beta_1} - \beta_1) \\ &= \frac{\sigma^2}{n} + (\bar{x})^2 Var(\hat{\beta_1}) \\ &= \frac{\sigma^2}{n} + \frac{\sigma^2 (\bar{x}) ^2} {SST_x}. \end{align}$

I believe this all works because since we provided that $\bar{u}$ and $\hat{\beta_1} - \beta_1$ are uncorrelated, the covariance between them is zero, so the variance of the sum is the sum of the variance. $\beta_0$ is just a constant, so it drops out, as does $\beta_1$ later in the calculations.

5) Use algebra and the fact that $\frac{SST_x}{n} = \frac{1}{n} \displaystyle\sum\limits_{i=1}^n x_i^2 - (\bar{x})^2$ :

V a r (β 0^) = σ 2 n + σ 2 ( x ¯ ) 2 S S T x = σ 2 S S T x S S T x n + σ 2 ( x ¯ ) 2 S S T x = σ 2 S S T x (1 n \sum i = 1 n x 2 i - (x ¯) 2) + σ 2 ( x ¯ ) 2 S S T x = σ 2 n - 1 \sum i = 1 n x 2 i S S T x

$\begin{align} Var(\hat{\beta_0}) &= \frac{\sigma^2}{n} + \frac{\sigma^2 (\bar{x}) ^2} {SST_x} \\ &= \frac{\sigma^2 SST_x}{SST_x n} + \frac{\sigma^2 (\bar{x})^2}{SST_x} \\ &= \frac{\sigma^2}{SST_x} \left( \frac{1}{n} \displaystyle\sum\limits_{i=1}^n x_i^2 - (\bar{x})^2 \right) + \frac{\sigma^2 (\bar{x})^2}{SST_x} \\ &= \frac{\sigma^2 n^{-1} \displaystyle\sum\limits_{i=1}^n x_i^2}{SST_x} \end{align}$

— M T
kaynak

There might be a typo in point 1; I think

var(β^) ${\rm var(\hat{\beta})}$ should read

β^ $\hat{\beta}$ .

— QuantIbex

You might want to clarify notations, and specify what

ui $u_i$ and

SSTx ${\rm SST}_x$ are.

— QuantIbex

ui $u_i$ is the error term and

SSTx $SST_x$ is the total sum of squares for

x $x$ (defined in the edit).

— M T

In point 1, the term

β1 $\beta_1$ is missing in the last two lines.

— QuantIbex

In point 2, you can't take

u¯ $\bar{u}$ out of the expectation, it's not a constant.

— QuantIbex