Tournament Selection Efficiency: An Analysis of the PGA TOUR s. FedExCup 1

Transcription

1 Tournament Selection Efficiency: An Analysis of the PGA TOUR s FedExCup 1 Robert A. Connolly and Richard J. Rendleman, Jr. June 27, Robert A. Connolly is Associate Professor, Kenan-Flagler Business School, University of North Carolina, Chapel Hill. Richard J. Rendleman, Jr. is Visiting Professor, Tuck School of Business at Dartmouth and Professor Emeritus, Kenan-Flagler Business School, University of North Carolina, Chapel Hill. The authors thank the PGA TOUR for providing the data used in connection with this study, Pranab Sen, Nicholas Hall and Dmitry Ryvkin for helpful comments on an earlier version of the paper and Ken Lovell for providing comments on the present version. Please address comments to Robert Connolly ( robert connolly@unc.edu; phone: (919) ) or to Richard J. Rendleman, Jr. ( richard rendleman@unc.edu; phone: (919) ).

2 Tournament Selection Efficiency: An Analysis of the PGA TOUR s FedExCup Abstract Analytical descriptions of tournament selection efficiency properties can be elusive for realistic tournament structures. Combining a Monte Carlo simulation with a statistical model of player skill and random variation in scoring, we estimate the selection efficiency of the PGA TOUR s FedExCup, a very complex multi-stage golf competition, which distributes $35 million in prize money, including $10 million to the winner. Our assessments of efficiency are based on traditional selection efficiency measures. We also introduce three new measures of efficiency which focus on the ability of a given tournament structure to identify properly the relative skills of all tournament participants and to distribute efficiently all of the tournament s prize money. We find that reasonable deviations from the present FedExCup structure do not yield large differences in the various measures of efficiency.

3 1. Introduction In this study, we analyze the selection efficiency of the PGA TOUR s FedExCup, a large-scale athletic competition involving a regular season followed by a series of playoff rounds and a finals event, where an overall champion is crowned. FedExCup competition began in Each year, at the completion of the competition, a total of $35 million in prize money is distributed to 150 players, with those in the top three finishing positions earning $10 million, $3 million and $2 million, respectively. 1 Research into selection efficiency highlights the importance of the criterion for assessing tournament properties. 2 Most who study tournament competition emphasize the probability that the best player will be declared the winner ( predictive power ) as the critical measure of tournament selection efficiency. Largely maintaining the focus of the selection efficiency literature on a single player, Ryvkin and Ortmann (2008) and Ryvkin (2010) introduce two additional selection efficiency measures, the expected skill level of the tournament winner and the expected skill ranking of the winner. They develop the properties of these selection efficiency measures in simulated tournament competition. While we use these efficiency measures in our work, we also develop three new measures of selection efficiency that evaluate the overall efficiency of a tournament structure, not just the the mean skill and mean skill rank of the first-place finisher and the expected finishing position of the most highly-skilled player. Much of the existing literature (e.g., Ryvkin (2010), Ryvkin and Ortmann (2008)) assumes a specific set of distributions (e.g., normal, Pareto, and exponential) to describe competitor skill and random variation in performance. In this paper, we integrate an empirical model of skill and random variation in performance with a detailed tournament simulation to explore the selection efficiency of FedExCup competition. We do not specify the matrix of winning probabilities as in some studies; instead, it is generated naturally from the underlying estimated distributions of competitor skill and random variation and the tournament structure itself. In the next section of the paper we describe the characteristics of FedExCup competition. We develop tournament selection efficiency measures in Section 3. We present an overview of the 1 See 2 See Ryvkin and Ortmann (2008) for an excellent recap of existing work along these lines. 1

4 statistical foundations of our work in Section 4, describe our simulation methods in Section 5, and present results and a discussion of practical implications of our work in Section 6. We summarize our findings in the final section. Appendix A describes the details of our simulation. 2. Characteristics of FedExCup Competition 2.1. Structure of FedExCup Competition Under current FedExCup rules, similar in structure to NASCAR s Sprint Cup points system, PGA TOUR members accumulate FedExCup points during the 35-week regular PGA TOUR season. 3 As shown in the Regular Season Points portion of Table 1, points are awarded in each regular season PGA TOUR-sanctioned event to those who make cuts using a non-linear points distribution schedule, with the greatest number of points given to top finishers relative to those finishing near the bottom. At the end of the regular season, PGA TOUR members who rank in FedExCup points are eligible to participate in the FedExCup Playoffs, a series of four regular 72-hole stroke play events, beginning in late August. In the Playoffs, points continue to be accumulated, but at a rate equal to five times that of regular season events. The field of FedExCup participants is reduced to 100 after the first round of the Playoffs (The Barclays), reduced again to 70 after the second Playoffs round (the Deutsche Bank Championship), and reduced again to 30 after the third round (the BMW Championship). At the conclusion of the third round, FedExCup points for the final 30 players are reset according to a predetermined schedule, with the FedExCup Finals being conducted in connection with THE TOUR Championship. The player who has accumulated the greatest number of FedExCup points after THE TOUR Championship wins the FedExCup FedExCup Competition Objectives It is clear that the objectives of FedExCup competition are multidimensional and complex. From the November 25, 2008 interview with PGA TOUR Commissioner Tim Finchem (PGA TOUR 3 The rules associated with FedExCup competition have been changed twice. Detail about the revisions is presented in Hall and Potts (2010). 4 A primer on the structure and point accumulation and reset rules may also be found at 2

5 (2008)), it is possible to identify a number of these dimensions. 1. The points system should identify and reward players who have performed exceptionally well throughout the regular season and Playoffs. As such, among those who qualify for the Playoffs, performance during the regular season should have a bearing on final FedExCup standings. 2. The Playoffs should build toward a climactic finish, creating a playoff-type feel, holding fan interest and generating significant TV revenue throughout the Playoffs. 3. The points system should be structured so that the FedExCup winner is not determined prior to the Finals. (In 2008, Vijay Singh only needed to show up at the Finals to win the FedExCup. This led to significant changes in the points structure at the end of the 2008 PGA TOUR season.) 4. The points system should give each participant in the Finals a mathematical chance of winning. We note that Bill Haas, the 2011 FedExCup winner and lowest-seeded player to ever win, was seeded 25th among the 30 players who competed in the Finals The points system should be easy to understand. Under the current system, any player among the top five going into the Finals who wins the final event (THE TOUR Championship) also wins the FedExCup. Otherwise, understanding the system, especially during the heat of competition, can be very difficult. We do not attempt to quantify the PGA TOUR s objectives, as summarized above. Instead, we evaluate the optimal selection efficiency of FedExCup competition based on two decision variables. The first is the Playoffs points multiple. Presently, Playoffs points are five times regular season points. This has a potential impact on Commissioner Finchem s objective points 1 and 2 above. Talking with PGA TOUR officials, we understand that the TOUR reassesses the FedExCup points structure at the end of every season and that this multiple is an important part of the discussion. Reflecting these discussions, we vary the multiple between 1 and 5 in integer increments. Our second decision variable is whether or not to reset accumulated FedExCup points at the end of the third Playoffs round. The present reset system is structured to satisfy objectives 3 and 4 and guarantee that any player among the top five going into the Finals who wins the final event will win the FedExCup (objective 5, at least in part). Although we are able to identify optimal competition structures evaluated in terms of our six efficiency measures, we find that the cost of deviating from optimal structure appears to be small. This finding suggests that the costs of the implicit constraints associated with the objectives listed above may not be high. 5 Although confusing, we adopt the convention used throughout sports competition that a low seeding or finishing position is a higher number than a high position. For example, in a 10-player competition, the highest seed is seeding position 1, while the lowest seed is position 10. 3

6 3. Measures of Efficiency In order to measure the selection efficiency of various FedExCup competition structures, we simulate entire seasons of regular PGA TOUR competition followed by four Playoffs rounds. In each simulation trial, we begin with a set of true player skills, or expected 18-hole scores. Throughout the regular season and Playoffs competition, each simulated score for a given player equals his expected score, as given by his true skill level, plus a residual random noise component. As the season progresses, and throughout the Playoffs, each player accumulates FedExCup points according to a defined set of rules as described in Section 4.1. We then estimate the efficiency of the FedExCup points system using the criteria described below Ryvkin/Ortmann Selection Efficiency Measures We use the following three measures of tournament selection efficiency, examined in detail by Ryvkin and Ortmann (2008) and Ryvkin (2010). 1. The winning (%) rate of the most highly-skilled player, also known as predictive power. 2. The mean skill level (expected 18-hole score) of the tournament winner. 3. The mean skill ranking of the tournament winner. Note that these three criteria focus on a single player, either the most highly-skilled player (predictive power) or the tournament winner. No weight is placed on the finishing positions of other players other than through their effect on the finishing position of the most highly-skilled player or the mean skill ranking or skill level of the tournament winner. We propose three new measures of selection efficiency that capture the ability of a given tournament format to properly classify all tournament participants according to their true skill levels, not just the player who is the most highly skilled, and to properly allocate tournament prize money. Even if the most highly-skilled player in FedExCup competition wins most of the time, the FedEx- Cup would surely lose credibility if the worst players in the competition could frequently finish near the top and win a significant portion of the prize money. Ideally, the FedExCup design would not only identify the single best player in the competition with high probability but would also 4

7 place players in finishing positions relatively close to their true skill rankings. As such, tournament prize money would generally be the highest for the most highly skilled and lowest for the lowest skilled and, therefore, players would be rewarded in relation to their true skill levels. Our final three measures of selection efficiency take the form of loss functions that reflect these tradeoffs Mean Squared Rank Error (L RE ) Consider a tournament of N players, i = 1, 2,..., N, ordered by true skill (or expected score) Y i, with Y 1 < Y 2,... < Y N. Let j(i) denote the tournament finishing position of player i. For example, if the most highly-skilled player finishes the tournament in 5th position, j(1) = 5. Then Y j(i) is the inverse transformation of true skill implied by player i s tournament finishing position, j(i), which we, henceforth, refer to as implied skill. Finally, let M j(i) denote the monetary prize to player i if he finishes the tournament in position j(i), with M 1 > M 2,... > M N. Thus, M i denotes what player i s prize would have been if his tournament finishing position had equalled his true skill ranking and M j(i) denotes player i s actual prize. Our first loss function, the mean squared ranking error, L RE, measures the extent to which the tournament errs in identifying the true skill rankings of the N tournament participants. L RE = 1 N = 2σ 2 i N (i j (i)) 2 i=1 ( 1 ρi,j(i) ), (1) where σi 2 = ( N 2 1 ) /12 is the variance of the ranking positions, i = 1, 2,..., N, and ρ i,j(j) is the Spearman rank order correlation of the true skill ranks, i, and tournament finishing positions, j(i). Thus, a tournament scheme that maximizes the Spearman rank, ρ i,j(j), will minimize the mean squared ranking error, L RE. 6 We note that L RE weights all ranking errors equally, regardless of the actual skill differences of the players who have been miss-ranked. Our final two efficiency measures reflect these differences. 6 We note that Spearman s footrule, another measure of ranking error, is equivalent to minimizing the sum of absolute ranking errors rather than squared ranking errors. 5

8 3.3. Mean Squared Skill Error (L SE ) The mean squared skill error is defined as follows: L SE = 1 N N ( ) 2 Yi Y j(i) i=1 = 2σ 2 Y (1 β Y ). (2) Here, σy 2 is the variance of true true player skill, and β Y is the OLS slope coefficient associated with a regression of true player skill Y i on implied player skill, Y j(i), or vice versa. When true skill rankings and tournament finishing positions are perfectly aligned, β Y = 1, and L SE = 0. Note that if Y is linear in skill rank, L SE = L RE. The mean squared skill error takes the form of a quadratic loss function, equivalent to the loss function underlying OLS regression and Taguchi s (2005) loss function used in quality control Mean Money-Weighted Squared Skill Error (L W SE ) Here we weight each value of ( ) 2 Y i Y j(i) in (2) by wi = M i / N M i, where M i is the dollar tournament prize to the player among N participants who finishes the tournament in position i. Thus, the mean money-weighted skill error is computed as follows: i=1 N ( ) 2wi L W SE = Yi Y j(i). (3) i=1 In this form, the greatest weight is given to implied skill errors for which the most money is on the line. Unlike the previous two loss functions as expressed in Equations (1) and (2), this expression cannot be simplified further without substantial restrictions on the functional form of the weighting function (and the consequent loss of generality) Mean Squared Error Deflators Inasmuch as the value of each mean squared error is difficult to interpret without a reference point, we deflate each by the corresponding variance of the variable whose error we are attempting to estimate (i.e., true skill rankings, true player skill and money-weighted true player skill.) Thus, 6

9 the deflators for the ranking error, skill error and weighted skill error are, respectively, D RE = ( N 2 1 ) /12, D SE = σy 2, and D W SE = N Yi 2w i ( N Y i w i ) 2. i=1 i=1 4. Optimizing FedExCup Competition 4.1. FedExCup Points Distribution and Accumulation The Regular Season Points section of Table 1 shows the distribution of regular season FedExCup points. WGC and Majors are allocated slightly more points than regular PGA TOUR events. Additional events, which are events held opposite of some WGC events and majors, are allocated half the points associated with each regular event finishing position. During the regular PGA TOUR season, players accumulate FedExCup points based on the regular season points schedule. At the end of the regular season, the top 125 players in accumulated FedExCup points qualify to participate in the Playoffs. Each participant in the Playoffs carries his accumulated FedExCup points into the Playoffs, but once in the Playoffs, FedExCup points are awarded and accumulated according to the schedule shown in the Playoffs Points section of the table. Note that the points distribution schedule for the first three rounds of the Playoffs is exactly five times the points distribution for regular PGA TOUR events conducted prior to the Playoffs. At the end of the first Playoffs round (The Barclays), only the top 100 players in accumulated FedExCup points are eligible to continue to the second round. After the second round (The Deutsche Bank Championship), only the top 70 players are eligible to continue to the third round. After the third round (The BMW Championship) only the top 30 players qualify for the FedExCup Finals (The TOUR Championship). Immediately prior to the Finals, points are reset for each of the Finals qualifiers according to the schedule shown in the Finals Reset column of the Playoffs Points section. Points are awarded during the Finals according to the schedule shown in the last column of Table 1. (Note that this is exactly the same distribution of points awarded to finishing positions 1-30 during the first three rounds of the Playoffs.) The points reset was put into place after the second year of FedExCup competition to ensure that no single player could have won the FedExCup prior to the Finals event and also to give each participant in the Finals a mathematical chance of winning the FedExCup. 7

10 4.2. What We Evaluate We limit our analysis of selection efficiency to the 125 players who qualify for the FedExCup Playoffs. Using all six efficiency measures, we evaluate the efficiency of the regular season points distribution system. 7 For this same group of 125 players, we then evaluate each of the six efficiency measures at the end of each round of the Playoffs in an attempt to determine if each successive round of the Playoffs improves selection efficiency for this group of 125 players. We also evaluate selection efficiency over all six measures at the end of every Playoffs round, but only for those players who qualify to play in each round. Our concern is whether the points system improves efficiency incrementally at the end of each round for remaining participating players. 5. Statistical Foundations 5.1. Data Our data, provided by the PGA TOUR, covers the PGA TOUR seasons. It includes 18- hole scores for every player in every stroke play event sanctioned by the PGA TOUR for years for a total of 151,954 scores distributed among 1,878 players. We limit the sample to players who recorded 10 or more 18-hole scores. The resulting sample consists of 148,145 observations of 18-hole golf scores for 699 PGA TOUR players over 366 stroke-play events. Most of the omitted players are not representative of typical PGA TOUR players. For example, 711 of the omitted players recorded just one or two 18-hole scores Player Skill Estimation Model We employ a variation of the Connolly and Rendleman (2008) model to estimate time-varying player skill and random variation in scoring for a group of professional golfers representative of PGA TOUR participants during the eight-year period As in Connolly and Rendleman (2008), we employ the cubic spline methodology of Wang (1998) to estimate skill functions and autocorrelation in residual errors for players with 91 or more scores. We employ a simpler linear 7 We take the regular season points distribution schedule as shown in Table 1 as given as well as the number of players who qualify for the Playoffs and each of its stages. 8 Generally, these are one-time qualifiers for the U.S. Open, British Open and PGA Championship who, otherwise, would have little opportunity to participate in PGA TOUR sanctioned events. 8

11 representation without autocorrelation, as in Connolly and Rendleman (2012), for players with 10 to 90 scores over the full sample period. 9 Simultaneously, we estimate fixed course effects and random round effects. We note that the model does not take account of specific information about playing conditions (e.g., adverse weather as in Brown (2011), pin placements, morning or afternoon starting times, etc.) or, in general, the particular conditions that could make scoring for all players more or less difficult, when estimating random round effects. Nevertheless, if such conditions combine to produce abnormally high or low scores in a given 18-hole round, the effects of these conditions should be reflected in the estimated round-related random effects. 10 When estimating player skill functions, we also obtain sets of player-specific residual scoring errors, denoted as θ and η. The θ errors represent potentially autocorrelated differences between a player s actual 18-hole scores, reduced by estimated fixed course and random round effects, and his predicted scores. The η errors represent θ errors adjusted for estimated first-order autocorrelation, and are assumed to be white noise. We refer to a player s skill estimate at a given point in time as an estimate of his neutral score, since estimated fixed course effects and random round effects have been removed. 6. Simulation of FedExCup Competition 6.1. Simulation Design We structure each of 40,000 simulation trials so that the composition of the player pool is similar to what one might observe in a typical PGA TOUR season. As such, we do not include all 699 players from the statistical sample in each trial. Instead, the number of players per trial varies between 9 We established the 91-score minimum in Connolly-Rendleman (2008) as a compromise between having a sample size sufficiently large to employ Wang s (1998) cubic spline model (which requires 50 to 100 observations) to estimate player-specific skill functions, while maintaining as many established PGA TOUR players in the sample as possible. The censoring of a sample in this fashion will have a tendency to exclude older players who are ending their careers in the early part of the sample and younger players who are beginning their careers near the end. If player skill tends to vary with age, such a censoring mechanism can create a spurious relationship, where mean skill across all players in the sample appears to be a function of time. (Berry, Reese and Larkey (1999) show that skill among PGA TOUR golfers tends to improve with age up to about age 29 and decline with age starting around age 36. Thus, ages tend to represent peak years for professional golfers.) To eliminate any type of age-related sample bias arising from a censored sample, we employ a 10-score minimum, rather than a 91-score minimum, and use simpler linear functions to estimate skill for those who recorded between 10 and 90 scores. 10 Interacted random round-course effects, with similar justification, are also estimated in Berry, Reese and Larkey (1999) and Berry (2001). We also estimate random round-course effects in our original 2008 model. However, we believe that the course component of a potential round-course effect is better modeled as fixed than random. 9

12 415 and 459 and reflects the actual number of players in the sample in each year, We also structure the simulations so that the simulated distributions of player skill (mean neutral score per round), scoring, and player tournament participation rates during the simulated regular season closely approximate those observed in the actual sample. Simulation details are provided in the Appendix Simulation Results Table 2 summarizes the simulation sample mean value of each of six efficiency measures at the end of the regular PGA TOUR season and at the end of each round of the Playoffs evaluated with respect to the 125 players who qualify for the Playoffs in simulated competition. Each of the six panels of Table 2 represents one of the six selection efficiency measures. For the efficiency measure shown in Panel A, First Place Rate of Best Player, higher values are better. In the remaining five panels, lower values indicate greater efficiency. Each efficiency measure is evaluated using a Playoffs points to regular season points weighting ratio that varies from 1 to 5. All efficiency measures shown in Panels D-F are the values computed from Equations 1-3, deflated by the corresponding variance of the variable whose error we are attempting to estimate. Efficiency for Playoff round 4 is evaluated without a points reset (NR) and with a reset (R). The points reset schedule is that given in Table 1 times weighting ratio divided by 5. We denote the end of the PGA TOUR regular season as Stage 0 and the end of Playoffs rounds 1-4 as Stages 1 through 4, respectively. In each panel, the best efficiency value is shown in bold for each stage 1-4. Except for the few efficiency measures shown in italics, the measures shown in bold are statistically superior in a one-sided test at the 0.05 level relative to all other values shown for the same stage. 11 Regardless of the points weighting or efficiency measure, efficiency improves during each stage of competition during Playoffs rounds 1-3 and from round 3 to round 4 when there is no points reset. Only in Panel C (mean skill rank of player in first place) and Panel D (mean squared rank error) are there any entries where the efficiency measure improves from round 3 to round 4 when points are reset after round 3. Thus, from the standpoint of pure mathematical efficiency, without regard to non-mathematical objectives that might lead to a reset being optimal, the 125 players in 11 We estimate statistical significance using 10,000 bootstrap samples drawn from the simulated data generated by 40,000 trials. 10

13 the Playoffs are generally ordered more efficiently after the third round of the Playoffs than after the final round with a reset. The optimal Playoffs points weight seems to vary by selection efficiency measure; weights of 3 and 4 generally provide the best efficiency. Although this suggests that the present Playoffs points weight of 5 may be too high, we argue in Section 6.3 that these differences may have little practical significance. Table 3 is organized similarly to Table 2. In contrast to Table 2 where the focus is on the 125 Playoffs participants, the focus in Table 3 is on selection efficiency computed incrementally for each round of the Playoffs for just those players participating in a specific round. Each value shown in the table reflects the mean value over 40,000 simulation trials of the ratio of the efficiency measure computed at the end of the stage to the efficiency measure computed at the beginning of the stage for stage participants only. In all but Panel A, a ratio less than 1 represents an improvement in efficiency from one stage of Playoffs competition to the next. Again, values shown in bold correspond to the best values per stage. Unless shown in italics, all other values in the same stage are significantly inferior at the 0.05 level than the best value shown in bold. Again, the points reset after the third Playoffs round tends to decrease selection efficiency; players who participate in the finals with a points reset tend to be ordered less efficiently after the final stage of competition than they were ordered prior to the final stage. However, a decrease in selection efficiency is not indicated for all measures. For example, there is unambiguous improvement in the mean squared rank error (Panel D) and an indication of improvement, depending upon the Playoffs points weighting, in Panel C (mean skill rank of player in first place) and in Panel E (mean square skill error). However in only one case (a weight of 2 in Panel D) is the efficiency value with a points reset better than that without. As in Table 2, optimal points weightings tend to vary by efficiency measure. Nevertheless, a weight of around 3 appears to produce the best efficiency measures, but in some cases, the current weight of 5 appears to be optimal Practical Significance Despite finding optimal values for the Playoffs points weighting and the decision whether to reset FedExCup points going into the final Playoffs round, we believe that the practical differences are 11

14 insignificant among efficiency outcomes based on optimal tournament design and those based on non-optimal design over the range of possible Playoffs design schemes that we consider. The entries in Table 4, which show the best and worst efficiency outcomes from the corresponding panels of Table 2 along with efficiency outcomes where the outcomes in each regular season and Playoffs event are determined randomly, provide support for this view. 12 (We show results for random outcomes using a Playoffs weight of 3 only. With random tournament outcomes, the weight has essentially no impact on any of the efficiency measures.) The outcomes in Panels A-C, based on the efficiency measures of Ryvkin and Ortmann, are the most straightforward to interpret. Panel A shows the rate at which the best player in the competition wins. At the end of the competition, the best and worst outcomes associated with regular (non-random) tournament competition fall between 63% and 44%. By contrast, with random tournament outcomes, the best player wins less than 1% of the time. Panel B shows the mean skill level (mean neutral score per round) of the first-place finisher. The best and worst outcomes at the end of regular tournament competition fall between and compared with when tournament outcomes are determined randomly. Panel C shows the mean skill rank of the player who finishes the competition in first place. Here the best and worst outcomes at the end of regular tournament competition fall between 3.17 and 4.58 compared with when outcomes are determined randomly. Clearly, on the basis of these three measures, (non-random) regular tournament competition dramatically improves each of the three efficiency measures over what might have otherwise been obtained with random tournament outcomes. Whether tournament design is technically optimal appears to be of second-order importance relative to the general structure of the competition itself. Each of the efficiency measures in Panels D-F are the values computed from Equations 1-3, respectively, deflated by the corresponding variance of the variable whose error is being estimated. If we further divide the values in Panel D by 2, we obtain ( 1 ρ i,j(i) ), where ρi,j(i) is the Spearman rank order correlation of the true skill ranks and tournament finishing positions. Best and worst values from regular tournament competition fall between 66% and 71%, which correspond to Spearman rank correlations of and By contrast, the value for the same efficiency 12 We maintain exactly the same simulation design as described in the appendix, but instead of basing tournament outcomes on scores, outcomes are based on random orderings of tournament participants, both before and after cuts. 12

15 measure corresponds to a Spearman rank correlation of 0.013, essentially zero. Clearly the tournament competition, whether optimally designed in terms of Playoffs point weights and the reset, significantly improves the rank ordering of participating players. If we divide the values in Panel E by 2, we obtain (1 β Y ), where β Y is the OLS slope coefficient associated with a regression of true player skill on skill implied by tournament finishing position. Best and worst values in Panel E fall between and 0.613, which correspond to slope coefficients of and With random ordering, the value for the same efficiency measure corresponds to a slope of 0.013, essentially zero. As in Panel D, the efficiency values from non-random competition, whether or not they reflect optimal tournament design, are substantially better than that obtained by a random ordering of players. The values for the money-weighted squared skill error, shown in Panel F, are not as readily interpreted. Nevertheless, best and worst values associated with regular competition fall between and compared with with random tournament outcomes, suggesting that achieving exactly optimal tournament design is not critical. Finally, it is clear that the reset substantially reduces efficiency as measured by the winning rate of the best player as summarized in the Panel A sections of Tables 2-4. With a points reset, efficiency, as measured at the end of the competition, is actually worse than at the end of the regular season (Stage 0). Although it is not so easy for the average person following professional golf to appreciate all the dimensions of the other efficiency measures, we suspect that most would have an intuitive feel for the skill levels of the best players in golf. 13 If the best players are not winning the FedExCup at a reasonably high rate, and in particular Tiger Woods over the period of our study, it isn t unreasonable to expect that the competition could lose credibility among those who follow professional golf. Otherwise, we see little cost in changing Playoffs points weights or the reset to satisfy PGA TOUR objectives that might not be easy to quantify. 13 In fact, the TOUR publishes player scoring averages and scoring averages adjusted for field strength throughout the PGA TOUR season. Although neither of these measures corresponds exactly to our mean neutral score, these averages, along with Official World Golf Rankings and other performance measures, make it relatively straightforward to identify the best golfers. 13

16 6.4. Competitiveness and Excitement It is clear from PGA TOUR Commissioner s November 25, 2008 interview that the PGA TOUR strives to create a competitive and exciting playoffs system, building toward a climactic finish, that will hold fan interest throughout. While aiming to reward players who have performed exceptionally well throughout the regular season, the TOUR does not want the FedExCup winner to be determined prior to the Finals. Thus, the TOUR is seeking to achieve a fine balance between player performance during both the regular season and Playoffs. This balance may not be easily quantified in terms of the tournament selection efficiency measures we have considered thus far. By construction, the points reset ensures that the ultimate winner of the FedExCup cannot be determined until the completion of the final Playoffs event. In the same simulations that underlie the results summarized in Tables 2 and 3, the winner of the competition would be determined prior to the Finals with probability under the present Playoffs points weighting scheme (weight = 5) if there were no reset. With Playoffs points weights of 1, 2, 3 and 4, the probabilities would be 0.143, 0.200, 0.252, and 0.299, respectively. Clearly, if the FedExCup winner were determined prior to the FedExCup Finals, there would be little fan interest in the final event, THE TOUR Championship, which occurs during the middle of the professional and college football seasons. As such, we believe that the PGA TOUR would view these probabilities as being unacceptably high. (If Tiger Woods, or equivalently, a player with his scoring characteristics, is excluded from the simulations, the probabilities for Playoffs weights of 1-5 would be 0.039, 0.050, 0.066, 0.087, and 0.113, respectively. 14 More detailed results for simulations that exclude Woods are provided in the online appendix.) Table 5 shows FedExCup winning percentages for 20 of the 125 players in the Playoffs. In Panel A, the players are ordered by their seeding positions, 1-20, at the beginning of the Playoffs. In Panel B, players are ordered by their skill rankings, 1-20, in relation to the field of Playoffs participants. We include an online appendix as a supplement to this paper, which shows results in both panels for players in all positions, Table 5 shows that without a reset and without giving more weight to FedExCup points earned 14 Since Woods was such a dominant player during the period on which our simulations are based, projections based on the inclusion of a player of Woods skill may be misleading, at least at the top position, for future periods where there may be no equivalently dominant player. 14

17 during the Playoffs relative to the regular season, the player who finished the regular season in first place would win the FedExCup 81.8% of the time. (If Woods is excluded from the simulations, this estimate falls to 78.6%.) Moreover, without a reset and with a Playoffs weight of 1, all players in seeding positions have less that a 1% probability of winning. (This is also the case if Woods in not included.) Clearly the winning rate of 81.9% for the top-seeded player and the very low winning rates associated with players seeded beyond position 4 are inconsistent with the TOUR s objectives. Even with a Playoffs weight of 5, the top-seeded player going into the Playoffs wins the FedExCup over 50% of the time with no reset, and no player beyond seeding position 9 has over a 1% chance of winning. (Without Woods, 33.5%, and no player past position 14 has more than a 1% chance of winning.) By contrast, regardless of the weight, the winning percentage rate of the top-seeded player is substantially lower with a reset, 35.6% to 39.2%, but not so low that his performance during the regular season goes unrewarded, and many more players have a legitimate chance to win. (Without Woods, 21.4% to 29.9%.) From the Stage 3 column of Table 2, we see that the most-highly skilled player in the competition is the number 1 Playoffs seed 56% to 60% of the time. Therefore, ignoring the remote possibility that this player would not make the Playoffs, the number 1 seed would be the most highly-skilled among the 125 Playoffs participants 56% to 60% of the time. 15 When this player is not the top seed, he is very likely be near the top. Panel B shows that with a reset, the winning percentage rate of the most highly-skilled player in the playoffs is greater than that of the number 1 seed. This suggests that even if the most highly-skilled player is not the number 1 seed, he still has a reasonably high chance of winning. By contrast, without a reset and with a Playoffs weight of 1, the most highly-skilled player does not win as often as the number 1 seed, but he does win more often with a Playoffs weight of 5. (These same relationships also hold when Tiger Woods is excluded from the simulations.) Table 6 shows percentage rates per FedExCup finishing position (through finishing position 10) for the top-10-seeded players going into the Finals. (The online appendix shows the same results for all 30 players in the Finals over all 30 possible finishing positions.) Panels A and B indicate that the percentage rates per finishing position are hardly affected by the Playoffs weighting scheme when there is a points reset going into the Finals. With either a weight of 1 or 5, the top 5 seeds all 15 Tiger Woods misses the Playoffs in 39 of 40,000 simulation trials. 15

18 have a reasonable chance to win, ranging from 5.5% to 45.8%. Although not shown, winning rates for players in seeding positions are all less than 1%. Also, we estimate that a player seeded in position 25 or worse, the same as Bill Haas position going into the 2011 Finals, would win the FedExCup only 0.26% of the time under the present system with a reset and Playoffs weight of 5. Thus, Haas win was clearly a very rare event. Panels C and D, show percentage rates per finishing position without a reset. With a Playoffs weight of 1, there is little remaining uncertainty about the ultimate winner and other top finishers; all are very likely to finish in the positions in which they started. This problem is mitigated somewhat with a Playoffs weight of 5. Nevertheless, the number 1 seed wins almost four out of five times. Interestingly, these same results tend to hold, but just to a slightly lesser extent, when Tiger Woods is not included in the FedExCup competition. Although not entirely evident, since not all players and finishing positions are shown, in panels A and B, except for the first and last seeds, each player s most likely finishing position is worse than his initial seed. In Panel C, where there is no points reset and points per Playoffs event are the same as those of regular season events, the most likely finishing position for almost every Finals participant is his Finals seeding position. (The only exceptions are for seeding positions ) This suggests that a competition scheme with no reset and no differential weighting of Playoffs and regular season events leaves very little drama and potential for position changes in the Finals. By contrast, even without a reset, but with a Playoffs points weight of 5, players in Finals seeding positions 4-28 are most likely to finish worse than they started (positions not shown in the table). Taken as a whole, we believe that the reset and the weighting of Playoffs points more heavily than those for regular season events plays a critical role in maintaining drama and potential fan interest throughout the Playoffs. From a pure efficiency standpoint, the reset tends to be suboptimal. Nevertheless, it is clear that without a reset, the PGA TOUR could not satisfy its objectives of conducting a meaningful regular season leading to playoffs with a climactic finish that both holds fan interest and has the potential to generate significant TV revenue. 16

19 7. Summary and Conclusions In this paper we introduce several new tournament selection efficiency measures and apply these measures and several existing measures in a systematic evaluation of the selection efficiency of the FedExCup competition run by the PGA TOUR. Our new measures are defined on the full range of tournament outcomes, not just the characteristics of the top finisher or most highlyskilled player. Using simulation, we evaluate the efficiency characteristics of specific alternative tournament structures. Our simulations show that relative to random selection, every variation on the FedExCup tournament selection method that we consider produces significant improvements in selection efficiency. Beyond this result, perhaps the most important regularity is that the points reset impairs tournament efficiency. On the other hand, one important aim of the points reset is to ensure that the competition is in doubt until the last moment. We show that the reset and weighting of Playoffs points more heavily than those of regular season events are critical elements in creating an exciting and dramatic set of Playoffs events. We acknowledge that our analysis of excitement and drama is much less scientific than our more direct mathematical assessment of tournament selection efficiency and believe that a more formal development of this aspect of competition could be an interesting area for future research. 17

20 Appendix Simulation Methodology A. FedExCup Regular Season and Playoffs Competition In simulating the accumulation of FedExCup points during the regular PGA TOUR season and Playoffs, we make the following assumptions. 1. Between 415 and 459 players participate for a full regular season prior to the FedExCup Playoffs in 35 4-round stroke play events players participate in each event. There is no picking and choosing of tournaments nor any qualifying requirements. 17 The probability that any single player participates in a regular season event reflects his actual participation frequency on the TOUR. 2. After the first two rounds of each regular season event, the field is cut to the lowest-scoring 70 players who then continue for two more rounds of tournament play FedExCup points are awarded for each tournament using the PGA TOUR Regular Season events points distribution schedule shown in Table 1, assuming each of the 35 tournaments is a regular PGA TOUR event rather than a major, a World Golf Championship event or an alternate event held opposite tournaments in the World Golf Championship series. 4. At the end of the 35-event regular season, the Playoffs begin with the top 125 players in FedExCup points participating in The Barclays, the first of four Playoffs events. The Barclays employs a cut after the first two rounds, with the lowest-scoring 70 players advancing to the final two rounds. At the completion of play, FedExCup points are added to those previously accumulated for each of the 125 Playoffs participants according to the schedule of Playoffs points shown in Table After The Barclays, the top 100 players in FedExCup points advance to the Deutsche Bank Championship. The Deutsche Bank employs a cut after the first two rounds, with the lowestscoring 70 players advancing to the final two rounds. FedExCup points are added to those previously accumulated for each of the remaining 100 Playoffs participants according to the schedule of Playoffs points shown in Table After the Deutsche Bank Championship, the top 70 players in FedExCup points advance to the BMW Championship, where there is no cut. FedExCup points are added to those previously accumulated for each of the remaining 70 Playoffs participants according to the schedule of Playoffs points shown in Table regular season events reflects the number of weeks of regular season PGA TOUR competition prior to the FedExCup Playoffs during In three of the 35 weeks, two PGA TOUR sanctioned events were played simultaneously, but no single player could have participated in the two events at the same time. Therefore, to simplify the simulations, we treat these weeks as if a single event were held. 17 A standard PGA TOUR event consists of 144 players. In the early and late parts of the PGA TOUR season, regular events tend to be reduced in size to 144 players due to limited daylight hours. The TOUR also conducts a few invitationals with smaller fields, along with a few smaller field select events, including tournaments in the World Golf Championship series. In addition, the Masters, one of the four majors, is a small field event, with 97 players participating in Generally, the lowest-scoring 70 players and ties make the cut in regular PGA TOUR events. It is almost certain that no ties will occur with our simulation methodology, but in the unlikely event that a tie does occur, the tie is broken randomly. 18

21 7. After the BMW Championship, the top 30 players in FedExCup points advance to THE TOUR Championship. 8. When simulating the present TOUR Championship structure, the number of FedExCup points for the 30 participating players is reset according to reset schedule shown in Table 1. Players are then awarded additional FedExCup points according to their finishing position in THE TOUR Championship, a four-round stroke play event with no cut, using the points distribution schedule for the Finals as shown in Table 1. The FedExCup winner is the player who has earned the most FedExCup points, not necessarily THE TOUR Championship winner. B. Player Selection Players are selected for regular season tournament participation using the following procedure. 1. A single year from our statistical sample, , is selected, with each year being selected exactly 40, 000/8 = 5, 000 times. 2. All players who actually participated in the selected year become the regular season player pool. 3. Players from the regular season pool are selected randomly for participation in each of the 35 regular season events, where the probability of any player being selected among the 144 tournament participants is equal to the proportion of total player weeks in which he actually participated in the year selected, assuming sampling without replacement. 19 C. Simulated 18-Hole Scoring The following procedure is used to generate 18-hole scores for players who could potentially compete in a given randomly selected PGA TOUR season. 1. A single mean skill level (mean neutral score) for each player is selected at random from the portion of his estimated spline-based skill occurring in the selected PGA TOUR season, This becomes the player s mean skill level for the entire season For each player k, a single θ residual is selected at random from among the entire distribution of n k θ residuals estimated in connection with his cubic spline-based skill function. 3. For each player k, 166 η residuals are selected randomly with replacement from among the entire distribution of n k η residuals estimated in connection with his cubic spline-based skill function. 4. Using the initial randomly selected θ residual, the vector of 166 randomly-selected η residuals, and player k s first-order autocorrelation coefficient as estimated in connection with his cubic spline fit, a sequence of 166 estimated θ residuals is computed. 19 In determining the extent of individual player participation on the TOUR, we use weeks played rather than tournament played, since, in a few weeks each year, two PGA TOUR-sanctioned events are held simultaneously. 20 We assume that the level of effort for each player throughout the entire regular season and Playoffs is the same as that reflected, implicitly, in his estimated skill function. 19

22 5. The 166 θ residuals are applied to player k s skill estimate to produce 166 simulated random 18-holes scores. The first 10 scores are not used in simulated competition but, instead, are generated to allow the first-order autocorrelation process to burn in. The next 156 are the scores required for a player who might be selected to play in every regular season tournament and who misses no cuts during the regular season (35 4 = 140) or during the four rounds of the Playoffs (4 4 = 16). We note that it is highly unlikely that all 156 scores would be used for any single player. 6. Starting with the 11th score, scores for each player k are applied in sequence as needed to simulate scoring during the regular season and Playoffs Suppose player 1 makes the cut in the first regular season event and player 2 missed the cut. If both are selected to play in the second regular season event, then simulated scoring in the second event will start with scores 15 and 13 for players 1 and 2, respectively. 20