Deriving an Optimal Fantasy Football Draft Strategy

Transcription

1 Stanford University CS 221 Artificial Intelligence: Principles and Techniques Deriving an Optimal Fantasy Football Draft Strategy Team Members: Alex Haigh Jack Payette Cameron Van de Graaf December 16, 2016

2 1 Introduction In this project, we design and implement an AI agent tasked with deriving the optimal player drafting strategy for fantasy football. At present, there is no established consensus on an optimal drafting strategy, and a variety of human experts advocate a range of differing strategies. Our guiding question is whether we can create an AI capable of outperforming both the ESPN auto-drafter and human agents? Our assumptions are that we are competing in a 10-team league with standard scoring and 16-player rosters, which is the format commonly used by websites like ESPN and Yahoo Sports. In terms of input/output behavior, our system ingests player rankings (based on each player s projected point total within a specific position) as well as the set of players remaining on the draft board, a fixed draft order, and the system s position in that order. We then model the drafting process as a game with 10 players, and seek to have our agent yield the optimal draft pick on each turn given the above information. For example, if we had already drafted the #3 QB, we would expect that our agent might prescribe drafting the #2 RB or #4 WR before drafting another player from the same position. There are a number of challenges inherent in our chosen problem. Perhaps the most formidable is the combination of an extremely large state space with relatively little input information (as we limited input data to the pre-season rankings). This means that every time the expectimax agent wants to draft, the agent goes through the highest projected player at every position, selects that player, and then simulates the rest of the draft with this pick. This, however, means that at every pick in this simulation that comes after, the agent must do the same thing. Since there are 6 draftable positions, the state space becomes intractable very quickly. This implies the necessity of a robust evalutation function, which helps, but is in turn complicated by the fact that projected score is only loosely correlated with final score. 2 Infrastructure In searching for an ideal dataset, we identified a few important characteristics. Since our agent uses preseason player projected point totals to construct our ordinal player rankings, the minimal requirement for our dataset was that it include full player ranking data for all National Football League teams; additionally, an extensive history (in terms of number of seasons covered) and easily-scrapeable format were desirable characteristics. We decided to use fftoday.com as our primary data source as 1

3 the web site had complete data going back to 2008, which was the longest history we were able to find, although the site proved fairly difficult to scrape. The following link has illustrative examples of the data for each year that we examined: 10&LeagueID= Our scraping script uses the Python Scrapy library to automatically download and format the data for each year, recording the player ranking and position, for our eventual use. More specifically, we scrape both projected points for every player and final points for every player, save them into two separate CSV files (since they come from separate links), merge them manually, and then employ a CSV parser to turn them into player objects. After storing the formatted data in.csv files, and with 8 years of rankings at approximately 650 ranked players per year, our final dataset size was around 5200 rows. Importantly, we used the first 6 years ( ) as training sets, while holding out 2014 and 2015 as dev and test sets, respectively. 3 Model 3.1 Formal definition Our task is modeled as a game, with the following definition: S start : Empty draft Actions(s): A list of possible players to draft Succ(s, a): Copy of a draft object where NFL player a has been drafted on Player(s) s team IsEnd(s): The drafting team is full U tility(s), Eval(s): Linear predictors that take projection data as input and output a final score (Weights were learned with TD learning for the evaluation function and linear regression by stochastic gradient descent for utility) P layer(s): The player currently drafting P layers: 10 agents with a pre-specified draft strategy Strategies: Oracle, Baseline, expectimax ( dumb evaluation), expectimax++ ( smart evaluation) 2

4 3.2 Baseline, Oracle, and Advanced Methods Our baseline is composed of a simple strategy that merely drafts the player with highest projected point total remaining, with the only constraint being to fill out the starting lineup first, followed by the bench. Our oracle is an agent that is fed the final point rankings from the end of the season in addition to the initial point rankings, and picks players based on this impossibly accurate information. Our actual trial methods range from a simple expectimax strategy with no learned evaluation function (simply use the projections), to an expectimax agent employing a simple linear regression function as an evaluation function, and finally an expectimax agent with a more complex evaluation function fitted with TD-learning. More details behind these approaches are presented in the following section. 4 Methods In order to tackle the game presented above, we implemented an expectimax strategy that runs according to the following procedure: similar to the oracle, we choose the expectimax player to play against 9 baselines. We simulate a draft of the expectimax player in every drafting positions for every year (as we did with the oracle), and we output mean final score for every drafting position. Below find the recurrence relation for our algorithm, where s represents the draft state, d represents depth, a 0 represents our agent, and a 1 through a 9 represent other agents in the draft (modeled as behaving under the baseline strategy): U tility(s) if IsEnd(s) Eval(s) ifd = 0 V max,baseline (s, d) = max a Actions(s) V max,baseline (Succ(s, a), d) P layer(s) = a 0 π baseline (s)v max,baseline (Succ(s, a), d) P layer(s) = a 1, a 2...a 8 π baseline (s)v max,baseline (Succ(s, a), d 1) P layer(s) = a 9 To project final (end-of-season) points based on the chosen roster, our first attempt ran using a linear predictor. The classifier ingests a processed final roster and projects the roster s final point total based on certain features of the roster. We then use this as our utility function in expectimax. To implement the linear classifier, we first needed to generate rosters by simulating 3

5 Position Equation QB rank RB ln rank WR ln rank TE ln rank K rank DEF rank Table 1: Regression equations to predict final output based on position rank 150 drafts per year from (training set), and 150 drafts for 2014 (development set). We did not incorporate any information/data from the 2015 projections, since 2015 was used as the test data for final analysis. For each simulated draft, we implemented a pseudorandom drafting algorithm - this consisted of each team choosing the top 3 players at each position and randomly picking one (with restrictions: once we need to complete starting lineup or hit the max number of possible players at a position). Our dataset has 6 years examined * 150 drafts/year * 10 teams/draft = 9000 rosters in the training set. There are 1500 rosters in the test set (2014). For each roster, we generated the final starting lineup and final points of that lineup. Finally, we ran SGD on the training data set with an objective value of regularized absolute deviation (we found that it was more robust to outliers than using squared loss and that regularization kept weights small) employing a feature extractor that examined several features of the team: The projected points of the team s i th best player at each position, e.g. WR1ProjectedPoints or RB3ProjectedPoints. A separate projection for each player s final output based on their projected position rank. Separately, we ran regressions to predict a player s final point total based on their final position ranking since they are very tightly correlated, and we used projected rank as a proxy for final rank is. (e.g. we know how many points the third best running back should have, and if Adrian Peterson is projected to be the third best running back, we get a feature called RB1ProjectionFromRank that has a corresponding value of our regression equation at rank 3). Equations shown below; we found that RBs, WRs, and TEs had negative logarithmic relationships between rank and final output, while QBs, DEFs, and Ks had linear ones. For each starter, their projected Value Above Replacement - that is, the number of points that they are projected to get minus the projection of the 4

6 worst starter at that position. We define the worst starter to be the player with the fewest total points that should still be on some team s starting lineup. For example, since there are 10 teams and each time starts 1 QB, the worst starter would be the 10th best quarterback. Finally, we had indicator features for the number of players at each position, with the template At least s (e.g. if a team had 3 running backs, it would have the following features: AtLeast1RB = 1, AtLeast2RBs = 1, AtLeast3RBs = 1). We found that these had little predictive value (weights were all set to zero), so we removed them from the final weight vector. For the final version of our algorithm, we used TD-learning to learn weights for our evaluation function (the predictor from before was for the utility function). The recurrences are given below; we used an ɛ-greedy exploration policy with ɛ = max(0, numiters ). totalnumiters V opt (roster) = w { φ(roster) p LegalP ositions(roster) π(roster; w) = arg max p LegalP ositions(roster) V opt (newroster; w) P r.ɛ P r.1 ɛ More specifically, newroster = old roster with the addition of the highest ranked player at position p, and LegalP ositions are the positions that the agent is allowed to draft (if it needs to fill certain starters before the draft ends, it won t be able to draft others). On each (roster, position, reward, newroster) : w w η (V opt (roster; w) (reward + V opt (newroster; w)) φ(roster) reward = Implied objective: { 0 len(newroster) 16 f inalp oints(startinglineup) otherwise 1 2 ((V opt(roster; w) (reward + V opt (newroster; w)) 2 From , we simulated 1000 drafts each with an epsilon-greedy exploration policy. The reward each time is 0; on the last turn of the draft, the reward is the final output of the final starting lineup. Additional features we included in this version of the evaluation function included: drafted in round, at least s in round, as well 5

7 as replacing empty starter slots (i.e. it s the 6th round and we have no QB, so for evaluation we substitute a baseline) with the 5th best player at that position on the draft board. 5 Results Table 1 shows the projected and final end-of-season point totals for all tested advanced methods compared against the baseline. Notably, the expectimax strategies perform better than baseline in both projected and final terms in 2014, with highest performance achieved by expectimax with our dumb evaluation heuristic for projected points and the utility evaluation function for final points. However, in 2015, results are more mixed, with expectimax strategies outperforming the baseline in pre-season projections but failing to surpass the baseline in final point totals. User Projected Final Projected Final Baseline Expectimax w/ all features Expectimax w/ Drafted in round Expectimax w/ s in round Expectimax w/ Utility Expectimax w/ dumb evaluation Table 2: Expectimax with range of evaluation functions vs. baseline The two charts below demonstrate the relative performance of each tested strategy over the different starting drafting positions. These figures are helpful in assessing whether the agent is particularly effective with any given drafting position (i.e. the first pick or the last pick), giving us increased granularity on how expectimax with a range of evaluation functions compares to the baseline strategy of merely drafting the highest ranked players first. The most important takeaway is that the expectimax agents tend to perform best relative to the baseline and oracle with mid to late-mid draft selection position. We see that the expectimax agents outperform the baseline by a substantial margin on projected points across positions, whereas the mid to late positions outperformance 6

8 Figure 1: Projected point performance by strategy over draft position is more notable in the final point tallies. As a quick aside, the oracle scores very low in projected point totals because it incorporates information from the future (end of season rankings) so it appears to make poor decisions viewed from the standpoint of the beginning of the season, but ends out substantially beating the other strategies in the end of season tallies. Figure 2: Final point performance by strategy over draft position 7

9 The following table presents results from our attempts to tune various hyperparameters of our linear classifier that was used for the utility function. All tests were conducted at 200 iterations, which was enough to achieve convergence. The minimal train and test error values were achieved with a λ value of.75 and and η value of.0001, though we note that η is clearly more important in determining error than λ. lambda 0 eta.001/numupdates train test lambda 0.25 eta.001/numupdates train test lambda 0.5 eta.001/numupdates train test lambda 0.75 eta.001/numupdates train test lambda 1 eta.001/numupdates train test Table 3: Hyperparamter tuning table 8

10 6 Discussion An error analysis of our work needs to take into account the somewhat unique nature of our data / project, which stems mainly from the limited quantity of data we had available with which to train and test our agent. In some sense, each year of draft history can be considered one data point, as each year is subject to its own random series of injuries, team chemistry, schedules, etc., interactions among with are less common across seasons. This observation leads directly into a second important point, which is that our agent s results demonstrate very high variability year to year. More explicitly, as mentioned in the results section above, all versions of the expectimax agent performed quite well against 2014, a year we did not train on; but in the second test year of 2015, the agent came in well below the baseline. This variance is a direct consequence of the paucity of independent data-points in our set, and to arrive at a more statistically robust conclusion on the performance of our agent we would need many more test seasons to smooth out the natural randomness from year to year. With the above points in mind, we note that the behavior of our agent corresponds well with how we would expect an intelligent drafter to behave in the first five rounds. It starts by drafting RBs, WRs, and TEs. This is because these positions have a negative log relationship between rank and score, so the higher ranked players are likely to far outperform lower ranked ones. Later in the draft, our bot picks many QBs, kickers, and DEFs. The reason this happens is that these positions have linear rather than negative log relationships (see above table) between rank and final output- so the probability of the 20th best defense being starter worthy is higher than the 40th or 50th running back breaking out. A further confounding factor is that while we made a simplifying assumption that fantasy football is based on yearly model we used vs the fact that fantasy football is a weekly game. Moreover, our model omits the essential other elements of fantasy football strategy including adding, dropping, and trading players. Nonetheless, when we examine our results for 2014, (dividing our total points gained over a season by 16 games in a standard season) we see that our best performing expectimax agent is able to outperform the 90 points per game yielded by the baseline algorithm and score 98 points per game on average. In an ESPN league, the average playoff team scores 95 a game, so this margin could be the difference between making playoffs and not. These results give us confidence that with further refinement, an AI draft- 9

11 ing agent could be a useful adjunct for human-derived drafting strategies in fantasy football. 7 Sources