Bayes, Oracle Bayes, and Empirical Bayes

Transcription

1 Bayes, Oracle Bayes, and Empirical Bayes

2 Empirical Bayes Inference Robbins (1951) Robbins (1956) Compound Decision Procedures Empirical Bayes Question How does empirical Bayes relate to Bayesian and frequentist inference? Intermediate framework: Oracle Bayes Bayes / Oracle Bayes / Empirical Bayes 2 / 37

3 A Familiar Empirical Bayes Setup Unknown prior g(θ) produces unseen parameters g(θ) θ 1, θ 2,..., θ N Each θ i independently gives observation x i N(θ i, 1), x = (x 1, x 2,..., x N ) Robbins Use all of x to estimate each θ i Bayes / Oracle Bayes / Empirical Bayes 3 / 37

4 Oracle Bayes Jiang and Zhang (2009) Oracle tells us the order statistic of the θ s, θ ord = ( ) θ (1), θ (2),..., θ (N) (but not which x i goes with which θ i ) We want to estimate ˆθ i that minimizes Expected Average Squared Error 1 N ( ) 2 EASE = E ˆθi θ i N Example: Two Towers Bayes / Oracle Bayes / Empirical Bayes 4 / 37 i=1

5 TWO TOWERS EXAMPLE: Oracle (Red), Observations (Black) g(theta) and fhat(x) theta and x Bayes / Oracle Bayes / Empirical Bayes 5 / 37

6 Using the Oracle ḡ(θ) empirical density of the θ s (probability 1 /N on θ (1), θ (2),..., θ (N) ) Bayes estimate (φ the standard normal density) ˆθḡ(x) = N / N θ (i) φ(x θ (i) ) φ(x θ (i) ) 1 1 Minimizes EASE among rules ˆθ(x) = t(x) Two Towers EASEḡ = MLE estimates ˆθ i = x i has EASE MLE = 1.0 Bayes / Oracle Bayes / Empirical Bayes 6 / 37

7 Empirical Bayes No Oracle Use x = (x 1, x 2,..., x N ) to form estimate ˆθ x (x) of Oracle rule ˆθḡ(x) Empirical Bayes regret EASE x EASEḡ = EBregret Two Towers EBregret = EASE x = = Bayes / Oracle Bayes / Empirical Bayes 7 / 37

8 The Frequentist Face of Empirical Bayes Oracle application is entirely frequentist: Minimizing EASE is a frequentist criterion Assumption θ i g(θ) is irrelevant! Question Is the 40% EASE reduction meaningful? Answer Yes for EASE, No for some other applications Bayes / Oracle Bayes / Empirical Bayes 8 / 37

9 Finite Bayes Inference θ i ind g(θ) and x i θ i ind N(θ i, 1) for i = 0, 1, 2,..., N We want posterior inference for θ 0 given x 0 and the sibling observations x = (x 1, x 2,..., x N ) If N then finite Bayes Bayes Next example: x 0 = 5, N = 50 Bayes / Oracle Bayes / Empirical Bayes 9 / 37

10 FINITE BAYES INFERENCE: 50 'sibling' observations to x0=5; what can we say about theta0? Frequency x0= x values Bayes / Oracle Bayes / Empirical Bayes 10 / 37

11 Red: likelihood N(x0=5,1); Green: Estimated prior from sibs, Black: Estimated posterior for theta0 posterior density prior x0=5 18% likelihood theta Bayes / Oracle Bayes / Empirical Bayes 11 / 37

12 Standard Bayes (Just one θ and x) Prior θ g(θ) Observe x p θ (x) [N(θ, 1)] Marginal density f(x) = X g(θ)p θ(x) dθ Bayesian inference x f(x) and θ x [ e g (x), v g (x) ] conditional expectation conditional variance For squared error: e g (x) is Bayes estimate ˆθ g (x) Bayes risk { (θ R g = E eg (x) ) } 2 = X v g(x)f(x) dx Rḡ = EASEḡ (= 0.563) Bayes / Oracle Bayes / Empirical Bayes 12 / 37

13 Tweedie s Formula (Efron, 2011) If x θ N(θ, 1): e g (x) = x + l (x) [l(x) = log f(x)] v g (x) = 1 + l (x) So ˆθ g (x) = x + l (x) and R g = [1 + l (x)] f(x) = 1 X [l (x)] 2 f(x) X Next: g(θ) = ḡ(θ), Two Towers Bayes / Oracle Bayes / Empirical Bayes 13 / 37

14 theta x: expectation e(x) + var(x)^.5, Two Towers example e(x)+ sqrt(v(x)) sqrt(v(x)) e(x) x value Bayes / Oracle Bayes / Empirical Bayes 14 / 37

15 Empirical Bayes Risk and Regret Idea Estimate ˆf(x) from data x = (x 1, x 2,..., x N ) ] ê(x) = x +ˆl (x) and ˆv(x) = 1 +ˆl (x) [ˆl(x) = logˆf(x) Empirical Bayes risk R(g, ê) = E { (ê(x) θ) 2} Lemma [ê(x) R(g, ê) = R g + eg (x) ] 2 f(x) dx X Bayes risk EBregret Only need f and its estimate ˆf Bayes / Oracle Bayes / Empirical Bayes 15 / 37

16 Estimating f(x): f-modeling Bin data x 1, x 2,..., x N y k = number of x s in bin k m k = midpoint of bin k ˆf = glm(y ns(m,df), poisson)$est/n Two Towers K = 44 bins (df = 5) Bayes / Oracle Bayes / Empirical Bayes 16 / 37

17 Histogram counts for the Two Towers data x[i], i=1:1500; K=44 bins ('bin[k]' in green) counts count y[k] midpoint m[k] x value Bayes / Oracle Bayes / Empirical Bayes 17 / 37

18 Fitted marginal density fhat(x) (1500) from glm(counts~ ns(x,5),poisson); green = true f(x) counts x value Bayes / Oracle Bayes / Empirical Bayes 18 / 37

19 Estimated conditional expectation e(x)=e{theta x} Two Towers, glm(y~ns(m,5),poisson); true e(x) in green EBregret=.007 ehat(x) estimated true x value Bayes / Oracle Bayes / Empirical Bayes 19 / 37

20 Data-based Formula for EBregret M = (ns(m,5), 1), kth row M k Ṁ, kth row dm k dm k ˆf = binned estimate of f(x) from glm(y M,poisson) Theorem E { EBregret } 1 { [M N trace diag (ˆf) ] 1 [Ṁ M diag (ˆf) } Ṁ] Decreases as 1 /N Doesn t account for bias of ê Bayes / Oracle Bayes / Empirical Bayes 20 / 37

21 The Poisson Case θ g( ) and x θ Poi(θ) [ pθ (x) = e θ θ x /x! ] f(x) = g(θ)p θ (x) dθ and θ x [ e g (x), v g (x) ] Robbins e g (x) = (x + 1)f(x + 1) f(x) and v g (x) = e g (x) [ e g (x + 1) e g (x) ] Bayes estimate ˆθg (x) = e g (x) with R g = f(x)v g (x) R [g, e(x)] = R g + f(x) [ e(x) e g (x) ] 2 X X Bayes / Oracle Bayes / Empirical Bayes 21 / 37

22 Butterfly Data Corbet in Malaysia, y 1 = 118 species trapped just one time each y 2 = 74 species trapped twice each... x y Bayes / Oracle Bayes / Empirical Bayes 22 / 37

23 Butterfly data: observed counts and fitted model glm(y~ns(x,5),poisson) red: zipf's law proportional 1/x 118 counts x value Bayes / Oracle Bayes / Empirical Bayes 23 / 37

24 Zipf s Law and Robbins Formula If f(x) = c/x then Robbins formula gives e g (x) = (x + 1)f(x + 1) f(x) = x So the MLE ˆθ i = x i is Bayes! Butterfly estimate R g = 5.24 and R(g, x) = 5.39 Bayes / Oracle Bayes / Empirical Bayes 24 / 37

25 Tweedie Poisson Estimation θ g( ) and x θ Poi(θ) f(x) = g(θ) ( e θ θ x /x! ) dθ λ = log θ is natural parameter E{λ x} = e g (x) = lgamma(x + 1) + l (x) Var{λ x} = v g (x) = lgamma(x + 1) + l (x) Same for truncated Poisson! Next: Butterfly data, ˆf from ns(df=5) Bayes / Oracle Bayes / Empirical Bayes 25 / 37

26 Tweedie for lambda: post expectation e(x) + v(x)^.5; Red line is (x,log(x)); Points from g modeling lambda Bayes risk =.363 MLE Risk.382 EB regret, x value Risk and Regret calcs for x in 1:20 Bayes / Oracle Bayes / Empirical Bayes 26 / 37

27 Empirical Bayes: g-modeling f-modeling: Never need to estimate prior g(θ) But only for criteria that depend just on f(x): EASE Tweedie Robbins false discovery rates... g-modeling (Efron, 2016): of prior g(θ) Direct parametric modeling Bayes / Oracle Bayes / Empirical Bayes 27 / 37

28 Parametric Models for g(θ) Exponential family log g(θ) = m(θ) β ( hidden GLM ) β a p-dimensional parameter; m(θ) say ns(θ,5) MLE ˆβ found by nonlinear maximization: β g β (θ) f β (x) y Mult(N, f β ) Trouble y Mult(N, f β ) not exponential family Advantage: Estimate g(θ x), Pr{θ > 2 x}, etc. Bayes / Oracle Bayes / Empirical Bayes 28 / 37

29 Estimated prior g(lambda) for the Butterfly data, lambda=log(theta); prior model M = ns(df=5) Bayes Risk & EBreg, lambda: theta: g(lambda) theta: lambda Bayes / Oracle Bayes / Empirical Bayes 29 / 37

30 Missing Species Problem Corbet How many new species in t more years of observation? Fisher, Good (Turing) Poisson process model f-modeling: E{#new} #old (Good and Toulmin, 1956) = Clever formula in terms of ˆf e θ 1 e θt/2 g(θ) dθ 1 e θ g-modeling: Substitute ĝ(θ) for g(θ) Bayes / Oracle Bayes / Empirical Bayes 30 / 37

31 Expected number new species seen in t years additional trapping; Red dots from Fisher Good Gaskins nonparametric formula # new species in two years additional years Bayes / Oracle Bayes / Empirical Bayes 31 / 37

32 A g-modeling Example gamnorm θ i Gamma 9 /3, x i N(θ i, 1), i = 1, 2,..., 3200 x = (x 1, x 2,..., x N ) for N = 15, 25, 50,..., 3200 g(θ) = ns(θ, 5) Next: posterior quantiles of θ 0 x 0 = 5 (finite Bayes inference as N increases) Bayes / Oracle Bayes / Empirical Bayes 32 / 37

33 POSTERIOR PERCENTILES of theta0 given x0=5 as the number of siblings increases; true gamnorm %iles at right limits log number siblings N Bayes / Oracle Bayes / Empirical Bayes 33 / 37

34 DTI Study 12 children, 6 dyslexic vs 6 normal controls N = brain voxels x i, x-value for ith voxel, x i N(θ i, 1) θ i = true effect size (= E{x i }) Bayes / Oracle Bayes / Empirical Bayes 34 / 37

35 DTI STUDY: x values for voxels, versus distance d from back of brain distance d from back of brain x value d=60,x=2.3 Bayes / Oracle Bayes / Empirical Bayes 35 / 37

36 g(theta0 x0=2.3) for decreasing sibling sets: (1) All (2) d in (3) (4) (5) only d=60 g(theta0 x0=2.3) x0= theta0 Bayes / Oracle Bayes / Empirical Bayes 36 / 37

37 References Efron, B. (2011). Tweedie s formula and selection bias. J. Amer. Statist. Assoc. 106: Efron, B. (2016). Empirical Bayes deconvolution estimates. Biometrika 103: Good, I. and Toulmin, G. (1956). The number of new species, and the increase in population coverage, when a sample is increased. Biometrika 43: Jiang, W. and Zhang, C.-H. (2009). General maximum likelihood empirical Bayes estimation of normal means. Ann. Statist. 37: Robbins, H. (1951). Asymptotically subminimax solutions of compound statistical decision problems. In Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability, UC Press, Robbins, H. (1956). An empirical Bayes approach to statistics. In Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, , Vol. I. UC Press, Bayes / Oracle Bayes / Empirical Bayes 37 / 37