Bayesball: A Bayesian Hierarchical Model for Evaluating Fielding in Baseball

Transcription

1 Bayesball: A Bayesian Hierarchical Model for Evaluating Fielding in Baseball Shane T. Jensen Department of Statistics, The Wharton School, University of Pennsylvania May 23, 2011 Joint Work with Kenny Shirley, James Piette and Abraham Wyner

2 Quantifying Fielding Performance in Baseball Overall goal: accurate evaluation of the fielding performance of each major league baseball player Official MLB Method: Errors and Fielding Percentage Only punishes for bad plays, does not reward for good plays or account for difficulty of successful plays Better Methods: Ultimate Zone Rating (M. Lichtman) and Probabilistic Model of Range (D. Pinto) divide field into large zones and tabulates of successful vs. unsuccessful plays for each fielder within zones Weakness of these methods: field not treated as single continuous surface Each zone/slice is large which limits ability to detect small differences between fielders

3 Baseball Info Solutions (BIS) Data Ball-in-play data available via Baseball Info Solutions 7 seasons (02-08) with balls-in-play (BIP) per year Three BIP types: 42% grounders, 33% flys, 25% liners Each BIP is mapped to a more resolute area than zones of previous methods BIP velocity information as ordinal category Flyballs Caught by CF Flyballs Not Caught by CF Y Coordinate Y Coordinate X Coordinate X Coordinate

4 Smooth Fielding Curves High-resolution data allows us to fit smooth fielding curves to the continuous playing surface Flyballs Liners Groundballs Y Coordinate Y Coordinate Y Coordinate X Coordinate X Coordinate X Coordinate Instead of non-parametric contours, we will use a hierarchical parametric model for fielding curves Hierarchical models allow for principled sharing of information within and between individual players

5 Bernoulli Successes and Failures The outcome of each play is either a success or failure: { 1 if the j th BIP hit to the i th player leads to out S ij = 0 if the j th BIP hit to the i th player leads to hit Observed successes and failures are modeled as Bernoulli realizations from an underlying probability: S ij Bernoulli(p ij ) Each p ij is a function of available data for that BIP: (x, y) ij location, velocity V ij and type of the BIP These probability functions will be smooth parametric curves that can vary between different players

6 Representation for Different BIP Types Two-dimensional curves needed for flys/liners: success depends on velocity, direction and distance to BIP One-dimensional curves needed for grounders: success depends on velocity, direction and angle to BIP Flyballs and Liners Grounders Y Coordinate Backward Forward CF Location at (0,324) Distance BIP Location Y Coordinate SS Location Grounder Trajectory Left Right θ Angle X Coordinate X Coordinate

7 Probit function for each smooth curve Probit regression used to model smooth curves for probability p ij of successfully fielding BIP j by player i Φ( ) = CDF of Normal distribution Probit function for fly-balls/liners: p ij = Φ(β i0 + β i1 D ij + β i2 D ij F ij + β i3 D ij V ij + β i4 D ij V ij F ij ) = Φ(X ij β i ) D ij = distance to BIP, V ij = vel, F ij = 1 if forward (vs. back) Probit function for grounders: p ij = Φ(β i0 + β i1 θ ij + β i2 θ ij L ij + β i3 θ ij V ij + β i4 θ ij V ij L ij ) = Φ(X ij β i ) θ ij = angle to BIP, V ij = velocity, L ij = 1 if left (vs. right)

8 Fitting Aggregate Model at each Position Standard probit regression software can be used to calculate MLE of coefficients β Can fit coefficients β + of aggregate model using data from all players at each position. Baseline probability of making a play for each infield position Probability B SS P 2B 1B Degrees Nice to compare between positions, but more interesting to compare between individual players at each position!

9 Individual Models for Grounders Can fit MLE coefficients β i of player-specific model using only data for player i Compare infielders based on MLE curves for grounders P(Success) for Everett, Jeter vs. average SS 1.0 P(Success) Average Jeter Everett rd Base SS Location 2nd Base 1st Base Degrees from SS

10 Individual Models for Fly/Liners Can again fit MLE coefficients β i of player-specific model using only data for player i Compare outfielders based on MLE curves for flys or liners

11 Problems with MLE Estimation Ideally want curve variability, not just MLE estimate Small samples for some players leads to unstable estimates of individual curves 2005 Centerfielder Curves Success Probability Individual Aggregate Distance Bayesian hierarchical model: shares information (and models variability) instead of separate estimation of player

12 Bayesian Hierarchical Model Player-specific coefficient vector now share common prior distribution across all players at a position: β i Normal(µ,Σ) µ is unknown mean vector and Σ unknown covariance matrix shared across all players Need prior for shared population parameters µ and Σ Assume a priori independence of β i components so Σ has off-diagonal elements of zero and diagonal elements σ 2 k Use recommended (Gelman, 2006) non-informative prior distribution: p(µ k,σ k ) 1 k = 0,...,4 Note that a priori independence will still lead to posterior dependence between β i components.

13 Model Implementation For each position and BIP-type, we need to estimate full posterior distribution of parameters p(β,µ,σ S, X) p(s β, X) p(β µ,σ) p(µ,σ) We estimate each posterior distribution using MCMC methods, specifically Gibbs sampling Iteratively sample from conditional distribution of each parameter set given other parameter values Data augmentation for Gibbs sampling with probit models: augment data with Z ij Normal(X ij β i, 1) and observe that P(S ij = 1) = Φ(X ij β i ) = P(Z ij 0) Z are a representation of our outcome data S that is now conjugate with the rest of our hierarchical model

14 Gibbs Sampling Steps 1 Sample Z ij N(X ij β i, 1) with truncation: sample Z ij until Z ij 0 (if S ij = 1) or Z ij < 0 (if S ij = 0) 2 Sample player coefficients β i from β i Normal [(Σ 1 + X i X i ) 1 (Σ 1 µ + X i Z i ), (Σ 1 + X i X i ) 1] 3 Sample population means µ k from ( m ) i=1 µ k Normal β ik, σ2 k m m k = 0,...,4 4 Sample population variances σk 2 from m (β σk 2 ik µ k ) 2 Inv Gamma (m 1) i=1, 2 2 k = 0,...,4

15 Results for Population Parameters As an example, examine µ for flyballs hit to CFs in 2005 Estimates and 95% Intervals for µ Intercept Distance Velocity Forward Posterior Coefficient Not surprisingly, coefficients for distance and velocity are negative: decreasing P(catch) with higher values Distance is clearly the predictor that explains the most variation in the outcome

16 Examining Shrinkage of Player Coefficients Hierarchical model shrinks each β i towards population µ Intercept Distance No Pooling No Pooling Partial Pooling Partial Pooling Complete Pooling Complete Pooling No Pooling Velocity No Pooling Forward Partial Pooling Partial Pooling Complete Pooling Complete Pooling No pooling = MLE, Partial pooling = Hierarchical model, Complete pooling = µ

17 Examining Shrinkage of Player Curves We can also examine shrinkage of player-specific curves (a) No Pooling (MLE)) (b) Partial Pooling (c) Complete Pooling P(Catch) P(Catch) P(Catch) Distance Distance Amount of shrinkage depends on sample size n i and variation in performance through X i and Z i Still heterogeneity between players even after shrinkage Distance

18 Numerical Summary of Overall Performance Beyond comparing curves between players, we can derive an overall numerical estimate of fielder performance SAFE: Spatial Aggregate Fielding Evaluation For each player, aggregate differences between individual curve (based on β i ) and overall curve (based on µ) Aggregation done by numerical integration over fine grid of values (1D grid for grounders, 2D grid for flys/liners) Can calculate SAFE for each sample from posterior distribution of β i, giving us the posterior mean and 95% posterior interval of SAFE for each player

19 Differential Weighting in SAFE Our full aggregation also weights grid points by BIP frequency, run value, and shared consequence (a) P(Success) for Jeter vs. Average (b) Density Estimate of Grounder Angle 1.0 P(Success) Average Jeter Density 0.0 3rd Base SS Location 2nd Base 1st Base 3rd Base SS Location 2nd Base 1st Base Degrees from SS (c) Run Consequence for Grounders Degrees from SS (d) Shared Responsibility of SS Runs Responsibility Fraction rd Base SS Location 2nd Base 1st Base 0.0 3rd Base SS Location 2nd Base 1st Base Degrees from SS Degrees from SS SAFE value: runs saved/cost of fielder vs. average

20 Results for Corner Infielders: Best/Worst Posterior SAFE values Ten Best 1B Player-Years Ten Best 3B Player-Years Name and Year Mean 95% Interval Name and Year Mean 95% Interval Doug Mientkiewicz, ( 2.8, 11.3 ) Marco Scutaro, ( 10.0, 16.6 ) Andy Phillips, ( 2.6, 11.4 ) Mark Bellhorn, ( 4.0, 17.1 ) Rich Aurilia, ( 2.7, 10.2 ) Hank Blalock, ( 4.2, 16.5 ) Albert Pujols, ( 3.1, 8.2 ) Sean Burroughs, ( 3.4, 14.2 ) Doug Mientkiewicz, ( 1.8, 9.1 ) David Bell, ( 1.7, 13.3 ) Albert Pujols, ( 1.9, 8.1 ) Scott Rolen, ( 1.9, 12.1 ) Kendry Morales, ( -0.5, 10.3 ) Hank Blalock, ( 1.4, 11.3 ) Ken Harvey, ( 1.5, 8.0 ) Damian Rolls, ( 0.1, 13.6 ) Howie Kendrick, ( -0.8, 9.6 ) Pedro Feliz, ( 0.5, 13.3 ) Albert Pujols, ( 1.0, 6.8 ) Joe Crede, ( 0.0, 15.8 ) Ten Worst 1B Player-Years Ten Worst 3B Player-Years Name and Year Mean 95% Interval Name and Year Mean 95% Interval Richie Sexson, ( -8.2, -1.9 ) Eric Munson, ( -12.4, -2.8 ) Robert Fick, ( -11.3, 2.0 ) Michael Cuddyer, ( -11.4, -2.9 ) Mo Vaughn, ( -9.7, -0.3 ) Michael Cuddyer, ( -14.1, -2.3 ) Dmitri Young, ( -9.9, 0.1 ) Garrett Atkins, ( -12.4, -2.4 ) Tony Clark, ( -11.7, -1.6 ) Fernando Tatis, ( -14.2, -2.0 ) Fred McGriff, ( -9.4, -2.8 ) Chone Figgins, ( -18.7, -1.4 ) Mike Jacobs, ( -9.4, -2.9 ) Travis Fryman, ( -15.2, -4.4 ) Ben Broussard, ( -10.4, -2.2 ) Joe Randa, ( -17.3, -2.8 ) Nomar Garciaparra, ( -11.1, -3.5 ) Ryan Braun, ( -17.4, -2.9 ) Jason Giambi, ( -13.4, -3.2 ) Jose Bautista, ( -17.4, -5.9 )

21 Results for Middle Infielders: Best/Worst Posterior SAFE values Ten Best 2B Player-Years Ten Best SS Player-Years Name and Year Mean 95% Interval Name and Year Mean 95% Interval Julius Matos, ( 12.4, 22.1 ) Pokey Reese, ( 12.0, 31.2 ) Erick Aybar, ( 10.0, 24.6 ) Adam Everett, ( 10.4, 27.4 ) Junior Spivey, ( 4.7, 27.1 ) Adam Everett, ( 9.0, 21.8 ) Tony Graffanino, ( 4.6, 27.6 ) Craig Counsell, ( 6.9, 21.1 ) Adam Kennedy, ( 1.7, 18.6 ) Jorge Velandia, ( 3.0, 24.0 ) Willie Bloomquist, ( 4.3, 17.8 ) Alex Cora, ( 3.0, 24.6 ) Jose Valentin, ( 4.2, 17.9 ) Alex Rodriguez, ( 3.5, 24.4 ) Chase Utley, ( 5.7, 17.5 ) Maicer Izturis, ( 3.8, 22.2 ) Chase Utley, ( 3.1, 17.7 ) Marco Scutaro, ( 4.0, 20.1 ) Craig Counsell, ( 5.3, 18.0 ) Brent Lillibridge, ( 5.0, 19.1 ) Ten Worst 2B Player-Years Ten Worst SS Player-Years Name and Year Mean 95% Interval Name and Year Mean 95% Interval Ronnie Belliard, ( -19.5, 2.6 ) Erick Almonte, ( -26.9, 2.3 ) Geoff Blum, ( -17.5, -1.7 ) Derek Jeter, ( -21.7, -5.8 ) Miguel Cairo, ( -17.9, -3.1 ) Michael Morse, ( -23.0, -4.5 ) Terry Shumpert, ( -22.2, 0.7 ) Damian Jackson, ( -30.6, -3.5 ) Roberto Alomar, ( -19.3, -4.6 ) Brandon Fahey, ( -22.4, -8.2 ) Enrique Wilson, ( -18.9, -6.2 ) Marco Scutaro, ( -22.0, ) Alberto Callaspo, ( -20.4, -4.5 ) Derek Jeter, ( -24.8, -6.4 ) Dave Berg, ( -25.1, -2.4 ) Michael Young, ( -23.6, -7.2 ) Luis Rivas, ( -20.9, -6.4 ) Josh Wilson, ( -26.5, -6.4 ) Bret Boone, ( -22.4, -8.1 ) Derek Jeter, ( -29.1, -9.2 )

22 Results for Outfielders: Best/Worst Posterior SAFE values Ten Best Left Fielders Ten Best Center Fielders Ten Best Right Fielders Name and Year Mean 95% Interval Name and Year Mean 95% Interval Name and Year Mean 95% Interval E Brown, ( 2.2, 27.9 ) J Michaels, ( 3.3, 32.5 ) G Matthews Jr., ( 5.7, 22.3 ) D Dellucci, ( 5.7, 20.4 ) C Figgins, ( 3.8, 31.2 ) D Mohr, ( 2.3, 28.0 ) R Johnson, ( 2.3, 21.0 ) J Hairston Jr., ( 0.3, 28.6 ) T Nixon, ( 3.3, 18.1 ) C Crisp, ( 4.1, 17.8 ) A Jones, ( 2.2, 20.7 ) G Matthews Jr., ( 2.6, 19.0 ) S Hairston, ( 1.1, 23.5 ) D Glanville, ( -3.1, 30.5 ) R Langerhans, ( 4.6, 19.3 ) S Podsednik, ( 6.1, 17.8 ) J Payton, ( 0.0, 17.8 ) T Nixon, ( 0.7, 19.4 ) M Byrd, ( 0.5, 22.7 ) J Edmonds, ( -0.5, 20.5 ) A Escobar, ( -0.1, 19.3 ) G Vaughn, ( 1.9, 16.9 ) J Gathright, ( -6.6, 25.0 ) A Ochoa, ( -2.3, 20.9 ) O Palmeiro, ( 0.8, 22.2 ) D Erstad, ( -1.2, 20.7 ) E Marrero, ( -0.9, 20.7 ) T Long, ( -0.8, 21.7 ) C Patterson, ( 1.9, 17.9 ) J Drew, ( -1.4, 20.8 ) Ten Worst Left Fielders Ten Worst Center Fielders Ten Worst Right Fielders Name and Year Mean 95% Interval Name and Year Mean 95% Interval Name and Year Mean 95% Interval K Mench, ( -19.4, -2.3 ) C Hermansen, ( -23.6, 4.3 ) G Kapler, ( -13.2, -1.6 ) K Mench, ( -14.9, -7.3 ) D Roberts, ( -21.0, 2.2 ) L Walker, ( -17.2, 1.2 ) A Piatt, ( -16.8, -6.9 ) R Ledee, ( -19.6, 0.5 ) J Guillen, ( -17.0, 0.7 ) L Berkman, ( -16.7, -8.2 ) K Griffey Jr., ( -24.4, -1.3 ) K Mench, ( -17.7, 0.7 ) M Ramirez, ( -19.1, -5.4 ) B Williams, ( -24.5, -3.1 ) E Kingsale, ( -13.1, -2.9 ) R Sierra, ( -16.2, ) S Green, ( -28.3, 2.8 ) W Pena, ( -17.6, 0.2 ) B Kielty, ( -16.7, -9.1 ) L Terrero, ( -29.4, 5.8 ) C Wilson, ( -22.1, -2.6 ) T Womack, ( -25.0, -5.0 ) B Williams, ( -23.4, -5.3 ) J Gonzalez, ( -16.4, ) L Berkman, ( -18.9, ) M Grissom, ( -34.2, -9.4 ) M Tucker, ( -21.8, -8.1 ) M Ramirez, ( -24.8, ) J Cruz, ( -36.2, -5.4 ) Sheffield, ( -21.6, -9.5 )

23 Sharing Information Over Time Want to share information across multiple years within each fielder (up to 7 seasons per player) Constant Ability Model: each season for a player is a realization from same underlying ability γ i β it Normal(γ i,τ 2 ) If players are consistent across time, our estimation of γ i should be more precise than seasonal β it s Still share information between players at the same position with common prior on γ i γ i Normal(µ,σ 2 )

24 Constant Ability Model Adding level to previous hierarchy to group player-seasons by player Position (µ) = average curve across that position. Player-Overall (γ i ) = average curve for player i. Player-Season (β it ) = average curve for player i in year t.

25 Constant Ability Model: Infielder Examples Infielder Grounders SAFE runs Chase Utley Luis Castillo Derek Jeter Overall Individual Years

26 Constant Ability Model: Outfielder Examples Outfielder Fly Balls Andruw Jones Jim Edmonds Manny Ramirez SAFE runs Overall Individual Years

27 Alternative: Moving Average By Age Model Assume that player ability is tied to an overall age-specific mean µ a for all players at that age. β ia Normal(µ a,σ 2 ) Information is shared across all seasons of players at that particular age.

28 Moving Average Model: Infielder Examples

29 Moving Average Model: Outfielder Examples

30 Model Comparison: Posterior predictive validation Predicted deviation: predicted probability of an out versus the actual outcome for all BIPs in a test dataset. 1 Split data into training ( 02-07) and testing ( 08). 2 Training data: get posterior samples from each model for each player 3 Testing data: calculate probability p ij of an out on each BIP using posterior samples 4 Predicted deviation is difference between predicted probability and actual outcome: D i = j y ij p ij, n i 5 Predicted deviation averaged across all posterior samples Compare models with Win% = proportion of players for which each model has smallest predicted deviations

31 Model Comparison: Results Moving average vs. Constant vs. Combined vs. MLE MA model performs best for most BIP x positions

32 Alternative: Autoregressive Model Add more individual specificity by modeling evolution over time within an individual fielder β it Normal(φ ait β i,t 1, τ 2 a it ) Player age a it incorporated through φ ait and τ ait parameters Allows autocorrelation within a player, but there is also age-dependent discounting that is shared across players through the φ a parameters. Can use dynamic linear model framework along with Kalman sampler implementation (current work)

33 Summary BIP data allows more detailed examination of differences between players Parametric approach: smooth probability function reduces variance of results by sharing information between all points near to a fielder Hierarchical structure also allows sharing of information between players at each position SAFE run value aggregates individual differences while weighting for BIP frequency, run value, and shared consequence between positions Lots of recent work on extending model to account for multiple seasons over time per player

34 Thank you! Paper: Bayesball: a Bayesian hierarchical model for evaluating fielding in major league baseball. Annals of Applied Statistics 3: SAFE Project Website: stjensen/research/safe.html Google search: shane jensen safe

35 Player Starting Positions Glossed over reality: we don t actually know where fielders are located when a particular BIP is hit! Each distance/angle is an estimated value, not a known value since starting location is not truly known Starting location for each position fixed at point with highest overall success probability Problematic when fielders are systematically different in their positioning e.g. 1B when there is a man on base Ideally, would build information about defensive shifts and men on base into model

36 Differences between Ballparks Current analysis does not take into account differences in the playing field for different parks Could impact both evaluation of infielders (turf vs. grass) and outfielders (different outfield shapes) Park-specific BIP densities could account for differences in shape but may have higher variance (less data)

37 Internal Validation: Posterior Predictive Simulation Can explore model fit with posterior predictive simulation Compare statistic of observed data to same statistic from data generated from model Repeatedly simulating datasets from model gives a posterior predictive distribution for observed statistic Look for systematic differences between data generated from model vs. observed data Substantial differences suggest aspect of data that model is failing to capture As an example, compare our hierarchical model (partial pooling) to complete pooling model with no heterogeneity between players

38 Posterior Predictive: Partial Pooling vs. Complete Pooling 500 datasets from partial & complete pooling models Statistic: % of flyballs caught for best CF minus worst CF Statistic for observed data indicated by vertical line Percentage Difference Distributions Partial Pooling Complete Pooling 15 Density Percentage Difference Partial pooling much better fit to observed data

39 External Validation: Comparison to UZR Decent overall agreement between SAFE and UZR Overall correlation between SAFE and UZR around 0.4 No gold standard for comparison, but can examine correlation between years (2002 vs. 2003) POS SAFE UZR DIFF 1B B B SS CF LF RF TOTAL Outfield is a big strength for SAFE, Infield has poorer performance (except for good 3B correlation)