Predicting NFL Game Outcomes. Paul McBride ECE 539 Professor Yu Hen Hu

Transcription

1 Predicting NFL Game Outcomes Paul McBride ECE 539 Professor Yu Hen Hu

2 Introduction Football has become an incredibly popular sport in America over the past 20 years or so. Every week, millions and millions of people tune in to watch the plethora of NFL games available to them. This has turned the NFL into a multibillion dollar corporation. This expansion of the NFL is also correlated with the rise popularity of sports news shows such as SportsCenter. On these programs, the analysts voice their opinions on who they think the winner of the games will be. These picks are largely based on personal opinion and not truly based on the data. I am looking to remove this human bias and find out truly what the data says about winning a football game. NFL football is an incredibly complex game. Sure, there are some stats you can count on to correlate with a win, such as total yards, but that is not always the case. A quarterback could have a bad game, a usually solid running back could cough up the ball in key positions; you really never totally know what is going to happen. These intangibles make predicting outcomes very challenging. But, based on the average gameplay of a team, it seems like using a large amount of data could be reliably used to smooth out the intangibles. Due to the NFL s incredible popularity, there exist a huge amount of game-by-game data online. Actually, the amount of data available is quite overwhelming. The challenging task is to pick stats that have a big impact on the gameplay. I decided to focus on offensive stats. The NFL is an incredibly offensive minded league and that s not to say that defense doesn t have an impact. It definitely does, but the play of a stellar quarterback has been known to be key to winning football games. I decided to get my data from a typical box score, or the team play stats per game. A standard box score has more than enough data to be effective. Box score contain offensive stats such as total yards, interceptions throw, turnovers, etc. So I will have more than enough useful stats to use. Just by taking a quick look at the numbers of a typical box score, one can tell that there is not just one telling stat on who won the game. For example, total yards is usually a good

3 indicator on who won the game the winning more often than not has more total yards than the losing team. But this just is not always the case. It s facts like this that lead me to believe that there does not exist a linear mapping method to choose the winner. And it s situations like this that are perfect for a neural network, more specifically a back-propagating multilayer perceptron. Work Performed Before any predicting could be performed, I had to decide on the data that I wanted to work with and to collect said data. As I alluded to in the introduction, the NFL is a very offensive minded game. Therefore, I wanted to analyze offensive stats to see if an efficient prediction network could come to fruition. Using my knowledge of football, I decided that the following stats would be good information for the neural network to learn from: 1. Homefield advantage: more often than not, the home team performs better in front of their home town fans. 2. Firstdowns: If a team has a high number of first downs, it means they were able to move the ball up and down the field. 3. Thirddown conversion percentage: If a team has a high thirddown conversion percentage, the offense is able to stay out on the field longer. 4. Rush attempts: This stat tracks how many times the team attempted to run the ball. 5. Rush yards: If the team has a high number of rushing yards, it usually complements the passing game quite nicely. 6. Pass attempts: This stat tracks how many times the team attempted to throw the ball. 7. Pass yards: If the team has a high number of pass yards, it usually complements the rushing game. 8. Total yards: If a team has a large amount of total yards, it mean there offense was working effectively.

4 9. Pass interceptions: This stat tracks the number of interceptions thrown by the quarter back. An interception turns the ball over to the other team s offense. 10. Fumbles lost: This stat tracks the number of fumbles lost by the team. A fumble picked up by the other team means it is their ball. 11. Turnovers: If a team has a high number of turnovers, it indicates that the offense was not able to stay out on the field. 12. Sacks: This stat tracks the number of times the quarter back was sacked. A sack results in a loss of yards. 13. Sack yards loss: This stats tracks the yards lost by the offense due to sacks. 14. Penalty yards: This stat tracks the amount of penalty yards lost by the offense. Penalties make it difficult for the offense to string together a number of plays to stay out on the field. I decided to collect data from the 2012 NFL season to train my network. I was able to get data from a site that has excel csv files available for download ( I chose the 2012 season because the brand of football played would be very similar to the 2013 season. After I downloaded the needed csv file, I uploaded the data into an excel workbook. I then, got rid of unneeded statistics. Since one team goes up against another team, I wanted to make the attributes of my feature vectors differential statistics. In other words, if the match up was Team A vs Team B, I subtracted the stats of Team B from the stats of Team A. If Team A had won that game, the outcome was 1. If Team B had won, the outcome was -1. Below is an example of a feature vector. This gave me 508 feature vectors to train my nueral network with. Each feature vector consisted of 15 attributes and one outcome attribute. Before I constructed a back-propagating multilayer perceptron, I wanted to have another classifier to compare the results to. I decided on a support vector machine. There are many configurations of a support vector machine that I could choose from, so for my baseline I decided to test three different svm s with different kernel functions: linear kernel function,

5 polynomial degree 3 kernel function, and a radial basis kernel function. For each of these I used constraint values of 1 and To get a true feel on how these support vector machines would perform in real life, I used 4-way cross validation to test the classification rate and to generate confusion matrices of the results. I give credit to Professor Hu s Matlab files for my training. However, the drivers that run the routines are all written by me. Below are the results I achieved using the support vector machines: Linear C = 1 Linear C = Polynomial C = Class % Polynomial C = RBF C = 1 RBF C = Class % As you can see, the support vector machine using a linear kernel with constraint value equal to 1 performed the best. I was very pleased with the results because predicting ~89% of games is quite the feat. Now that I had my baseline classification percentage, it was time to decide what structures of multilayer perceptron to use. The reason I decided to use a back-propagating neural network was for its great flexibility and that the back-propagation provides supervised learning. Since there are many ways an NFL game can go, supervised learning was an attractive feature. And since it s flexible, there are many different parameters that I can try to use to increase the classification rate of the network. Again the algorithm and code used to train the network is based off of the Matlab file supplied by Professor Hu on the course website. But the drivers and the data preparation were all written by me. Once again, I decided to use 4-way cross validation to mimic how the machines would be used in real use.

6 Right off the bat, I did not really know what to expect so I tried a variety of different inputs. Below are a few example of my initial results: # hidden layer = 5 mu=0 mu=.8 runs\alpha avg # hidden layer = 2 mu=0 mu=.8 runs\alpha avg # hidden layer = 2 runs\alpha mu=

7 avg runs\# in hidden layer alpha=.01 mu= avg Right off the bat the results were discouraging. The best it seemed that I could do was ~85% classification rate. While that is actually pretty good, It was not better than the linear support vector machine that I had previously trained. It was then that I decided to preprocess the data by running a singular value decomposition. The follow code snippet is an example of my data preprocessing: [ua,sa,va] = svd(data(:,1:15),0); m=11; u_vec1 = ua(:, 1:m); vec1 = u_vec1 * u_vec1'; svddata = [vec1*data(:, 1:15) data(:,16:17)]; The challenge doing this was deciding how many of the principal vectors to process my data with (the choice of variable m in the code snippet). I just decided to give a couple configurations a try:

8 # hidden layer = 5 mu=0 alpha =.01 runs\# of SVD prince vectors avg Right away, I could tell that the results were looking more promising. I was able to predict about 86% of the games without much tinkering of the input variables. Through a great deal of analysis, I was able to raise the prediction classification rate to 88% using 11 singular value decomposition vectors. Below are the two most successful configurations: # hidden layer = 5 mu=.02 alpha =.007 SVD prince vectors # hidden layer = 2 mu=.02 alpha =.006 SVD prince vectors Configuration #1 ( % classification rate): 1) # of total layers = 3

9 2) # of hidden neurons = 5 3) Momentum =.02 4) Learning rate =.007 5) # of SVD principal vectors = 11 Configuration #2 ( % classification rate): 1) # of total layers = 3 2) # of hidden neurons = 2 3) Momentum =.02 4) Learning rate =.006 5) # of SVD principal vectors = 11 I was very pleased with the results of preprocessing the data with singular value decomposition. The classification rate I was able to produce with the back-propagating multilayer perceptron was on par with the support vector machine! Now that I had my network of choice, I wanted to utilize it to do some predictions. I decided on week 15 of the 2013 NFL season. I collected the all 32 team s relevant statistical averages and compiled feature vectors from the head to head matchups. Since the classification rate of mlp configuration #1 is a little better, I used it to make my predictions.

10 MLP: SVM: Broncos Falcons Buccs Giants Vikings Dolphins Jags Colts Browns Raiders Panthers Cowboys Titans Rams Steelers Lions Chargers Redskins 49ers Seahawks Eagles Pats Bills Texans Bears Chiefs Jets Packers Cards Saints Bengals Ravens Broncos Falcons Buccs Giants Vikings Dolphins Jags Colts Browns Raiders Panthers Cowboys Titans Rams Steelers Lions Chargers Redskins 49ers Seahawks Eagles Pats Bills Texans Bears Chiefs Jets Packers Cards Saints Bengals Ravens Green represents a win. The MLP and the SVM predicted that game outcomes very similarly. They only differed on the outcomes of the Falcons vs. Redskins game and the Cowboys vs. Packers game. This can be explained because in those two matchups both teams are very comparable. For the most part, and using my human bias, most of the predictions seem very reasonable. There is only one game that is very questionable and that is the Broncos versus the Chargers. The Broncos are the heavy favorite and virtually every sports news pundit has picked them to win. But that is why I chose to do this. I wanted to remove the human bias. The Results

11 Actual Game Outcomes MLP Week 15 Results SVM Week 15 Results Broncos Chargers Broncos Chargers Broncos Chargers Falcons Redskins Falcons Redskins Falcons Redskins Buccs 49ers Buccs 49ers Buccs 49ers Giants Seahawks Giants Seahawks Giants Seahawks Vikings Eagles Vikings Eagles Vikings Eagles Dolphins Pats Dolphins Pats Dolphins Pats Jags Bills Jags Bills Jags Bills Colts Texans Colts Texans Colts Texans Browns Bears Browns Bears Browns Bears Raiders Chiefs Raiders Chiefs Raiders Chiefs Panthers Jets Panthers Jets Panthers Jets Cowboys Packers Cowboys Packers Cowboys Packers Titans Cards Titans Cards Titans Cards Rams Saints Rams Saints Rams Saints Steelers Bengals Steelers Bengals Steelers Bengals Lions Ravens Lions Ravens Lions Ravens Looking at the results, it seems as if both predicting machines did very poorly. And yes, it is true that this week they did. But if we look closer at week 15, we see that there were many games that are considered upsets. And this just shows the unpredictability of the NFL. The intangibles are very hard to predict. The upsets: 1) The Vikings over the Eagles: The usually explosive Eagles offense did not perform up to their usual standard. But even more unexpected was the offensive output of the Vikings. This game epitomizes the unpredictability of the NFL. 2) The Dophins over the Patriots: The usually high-powered offense of the Patriots had an off day. This could be because the play-making ability of the injured tight Rob Gronkowski was absent from the field. 3) The Rams over the Saints: Drew Brees and the Saints were let down by the atrocious effort in the running game. The rest of the games were just too close to call.

12 Future Attempts If I were to do this again in the future, I would add defensive stats to the feature vector. While the NFL is a very offensive oriented league, the defense definitely does play a role in the outcome of the games. While 88% predictability rate is very high, I believe that adding a defensive element to the feature vectors could push the classification rate into the 90 percentile. References This site had reliable csv files full of nfl stats NFL.com, ESPN.com,

13 Matlab Code 1. final_svm_classify.m This matlab code trained and tested via 4-way cross validation different configurations of a support vector machine. clear all; close all load TeamStatsDifferentials2012.txt data = TeamStatsDifferentials2012; for i=1:15 data(:,i) = translate(data(:,i), 0, 10); train = zeros(381,16,4); test = zeros(127,16,4); data1 = data(1:127, 1:16); data2 = data(128:254, 1:16); data3 = data(255:381, 1:16); data4 = data(382:508, 1:16); train(:,:,1) = [data1;data2;data3]; train(:,:,2) = [data1;data2;data4]; train(:,:,3) = [data1;data3;data4]; train(:,:,4) = [data2;data3;data4]; test(:,:,1) = data4; test(:,:,2) = data3; test(:,:,3) = data2; test(:,:,4) = data1; temp = ['linear '; 'linear '; 'polynomial';... 'polynomial'; 'rbf '; 'rbf ']; funct = cellstr(temp); C = [1, , 1, , 1, ]; options = optimset('maxiter',100000); ConfMatrix = zeros(2,2,6); ClassRate = zeros(1,6); for i=1:6 TN=0; FN=0; FP=0; TP=0; for j=1:4 model=svmtrain(train(:,1:15,j), train(:,16,j), 'kernel_function',...

14 char(funct(i)),'options',options,'boxconstraint',... C(i), 'method', 'QP'); tested=svmclassify(model,test(:,1:15,j)); for k=1:127 if test(k,16,j)==1 if tested(k,1)==1 TP = TP+1; else FN = FN+1; else if tested(k,1)==1 FP = FP+1; else TN = TN+1; ConfMatrix(1,1,i) = ConfMatrix(1,1,i) + TN; ConfMatrix(2,1,i) = ConfMatrix(2,1,i) + FN; ConfMatrix(1,2,i) = ConfMatrix(1,2,i) + FP; ConfMatrix(2,2,i) = ConfMatrix(2,2,i) + TP; ClassRate(1,i) = (TN+TP)/508; 2. final_svm_classify_predict.m This matlab script trained a linear svm and tested the prediction data. clear all; close all load TeamStatsDifferentials2012.txt; data = TeamStatsDifferentials2012; load PredictionData.txt; testdata = PredictionData; for i=1:15 data(:,i) = translate(data(:,i), 0, 10); testdata(:,i) = translate(testdata(:,i), 0, 10); C = [1, , 1, , 1, ]; options = optimset('maxiter',100000); model=svmtrain(data(:,1:15), data(:,16), 'kernel_function',... 'linear','options',options,'boxconstraint',... C(1), 'method', 'QP'); tested=svmclassify(model,testdata(:,1:15)); 3. mybp_svm.m

15 This matlab script trained and test via 4-way cross validation back-propagating mlps. This input came in through a text file. clear all, close all load TeamStatsDifferentials2012_mlp.txt data = TeamStatsDifferentials2012_mlp; for i=1:15 data(:,i) = translate(data(:,i), 0, 10); [ua,sa,va] = svd(data(:,1:15),0); ConfMat = zeros(2,2,1); ClassRate = zeros(9,3); file = fopen('mlpinput2.txt', 'r'); for b=1:10; m=11; u_vec1 = ua(:, 1:m); vec1 = u_vec1 * u_vec1'; svddata = [vec1*data(:, 1:15) data(:,16:17)]; data1 = svddata(1:127, :); data2 = svddata(128:254, :); data3 = svddata(255:381, :); data4 = svddata(382:508, :); traindata(:,:,1) = [data1;data2;data3]; traindata(:,:,2) = [data1;data2;data4]; traindata(:,:,3) = [data1;data3;data4]; traindata(:,:,4) = [data2;data3;data4]; testdata(:,:,1) = data4; testdata(:,:,2) = data3; testdata(:,:,3) = data2; testdata(:,:,4) = data1; line = fgetl(file); input = str2num(line); for c=1:8 ConfMat(:,:,b) = 0; for a=1:4 mybpconfig_svm(input, traindata(:,:, a), testdata(:,:,a)); % configurate the MLP network and learning parameters. load mlpconfig.mat % BP iterations begins while not_converged==1, % start a new epoch % Randomly select K training samples from the training set.

16 M+N [train,ptr,train0]=rsample(train0,k,kr,ptr); % train is K by z{1}=(train(:,1:m))'; % input sample matrix M by K d=train(:,m+1:mn)'; % corresponding target value N by K % Feed-forward phase, compute sum of square errors for l=2:l, % the l-th layer u{l}=w{l}*[ones(1,k);z{l-1}]; % u{l} is n(l) by K z{l}=actfun(u{l},atype(l)); error=d-z{l}; % error is N by K E(t)=sum(sum(error.*error)); % Error back-propagation phase, compute delta error delta{l}=actfunp(u{l},atype(l)).*error; % N (=n(l)) by K if L>2, for l=l-1:-1:2, delta{l}=(w{l+1}(:,2:n(l)+1))'*delta{l+1}.*actfunp(u{l},atype(l)); perturbation % update the weight matrix using gradient, momentum and random for l=2:l, dw{l}=alpha*delta{l}*[ones(1,k);z{l-1}]'+... mom*dw{l}+randn(size(w{l}))*0.005; w{l}=w{l}+dw{l}; % display the training error %bpdisplay; satisfied, % Test convergence to see if the convergence condition is cvgtest; t = t + 1; % increment epoch count % while loop %disp('final training results:') if classreg==0, [Cmat,crate]=bptest(wbest,tune,atype); elseif classreg==1, SS=bptestap(wbest,tune,atype), if testys==1, %disp('apply trained MLP network to the testing data. The results are: '); if classreg==0, [Cmat,crate,cout]=bptest(wbest,test0,atype,labeled,N); if labeled==1, %disp('confusion matrix Cmat = '); disp(cmat); %disp(['classification = ' num2str(crate) '%']) elseif labeled==0,

17 % print out classifier output only if there is no label disp('classifier outputs are: ') disp(cout); elseif classreg==1, SS=bptestap(wbest,test0,atype), ConfMat(:,:,b) = ConfMat(:,:,b) + Cmat; ClassRate(c,b) = (ConfMat(1,1,b)+ConfMat(2,2,b))/508; ClassRate(9,b) = mean(classrate(1:8,b)); 4. mybp_config.m This matlab script takes in input to set up the mlp, training data, and testing data. function mybpconfig( input, train, test ) train0 = train; [Kr,MN]=size(train); M= MN-2; N=MN-M; testys=1; test0 = test; [Kt,MNt]=size(test0); if MNt == MN, labeled=1; else labeled=0; scalein= input(2); [tmp,xmin,xmax]=scale([train0(:,(1:m)); test0(:,(1:m))],-5,5); train0(:,(1:m))=tmp((1:kr),:); test0(:,(1:m))=tmp((kr+1):(kr+kt),:); L1 = input(3); L=L1+1; n(1) = M; n(l) = N; j = 4; for i=2:l-1, n(i)= input(j); w{i}=0.001*randn(n(i),n(i-1)+1); % first column is the bias weight dw{i}=zeros(size(w{i})); % initialize dw j = j + 1; w{l}=0.005*randn(n(l),n(l-1)+1); % first column is the bias weight dw{l}=zeros(size(w{l}));

18 % choose types of activation function % default: hidden layers, tanh (type = 2), output layer, sigmoid (type = 1) % default parameter T = 1 is used. atype=2*ones(l,1); atype(l)=1; % default %disp('by default, hidden layers use tanh activation function, output use sigmoidal'); chostype= input(j); %if isempty(chostype), chostype=0; if chostype==1, disp('============================================================='); disp('activation function type 1: sigmoidal'); disp('activation function type 2: hyperbolic tangent'); disp('activation function type 3: linear'); for l=2:l, atype(l)=input(['layer #' int2str(l) ' activation function type = ']); % next load a tuning set file to help determine training errors % or partition the training file into a training and a tuning file. %disp('enter 0 (default) if a pattern classification problem, '); classreg= input(j+1); %if isempty(classreg), classreg=0; %disp('============================================================='); % msg_1=[... % 'To estimate training error, choose one of the following: ' % '1 - Use the entire training set to estimate training error; ' % '2 - Use a separate fixed tuning data file to estimate training error;' % '3 - Partition training set dynamically into training and tuning sets;' % ' (This is for pattern classification problem) ']; % disp(msg_1); %chos=input('enter your selection (default = 1): '); chos= input(j+2); %if isempty(chos), chos=1; if chos==1, tune=train0; elseif chos==2, dir; tune_name=input(' Enter tuning filename in single quote, no file extension: '); eval(['load ' tune_name]); tune=eval(tune_name); eval(['clear ' tune_name]); % scale the tuning file feature vectors and output as it is newly loaded. if scalein==1, % scale input to [-5 5] [tune(:,1:m),xmin,xmax]=scale(tune(:,1:m),-5,5); elseif chos==3, % partition the training file into a training and tuning % according to a user-specified percentage prc=input('percentage (0 to 100) of training data reserved for tuning: ');

19 [tune,train0]=partunef(train0,m,prc); [Kr,MN]=size(train0); % this train0 is only a subset of original train0 and hence % Kr must be updated. [Ktune,MN]=size(tune); % scaling the output of training set data % normally the output will be scaled to [outlow outhigh] = [ ] % for sigmoidal activation function, and [ ] for hyperbolic tangent % or linear activation function at the output nodes. % However, the actual output of MLP during testing of tuning file or testing % file will be handled differently: % a) Pattern classification problem: since we are concerned only with the maximum % among all output, the output of MLP will not be changed even it ranges only % between [outlow outhigh] rather than [0 1] % b) Approximation (regression) problem: the output of MLP will be scaled back % for comparison with target values % disp('============================================================='); % disp('output from output nodes for training samples may be scaled to: ') % disp('[ ] for sigmoidal activation function or '); % disp('[ ] for hyperbolic tangent or linear activation function '); %scaleout=input('=enter 1 (default) to scale the output: '); scaleout= input(j+3); %if isempty(scaleout), scaleout=1; if atype(l)==1, outlow = 0.2; elseif atype(l)==2 atype(l) == 3, outlow = - 0.8; outhigh=0.8; if scaleout==1, [train0(:,m+1:mn),zmin,zmax]=scale(train0(:,m+1:mn),outlow,outhigh); % scale output % scale the target value of tuning set [tune(:,m+1:mn),zmin,zmax]=scale(tune(:,m+1:mn),outlow,outhigh); % scale target value of testing set if available if testys==1 & labeled==1 % if testing set specifies output [test0(:,m+1:mn),zmin,zmax]=scale(test0(:,m+1:mn),outlow,outhigh); % now, we have a training file and a tuning file % learning parameters %disp('============================================================='); %alpha=input('learning rate (between 0 and 1, default = 0.1) alpha = '); alpha= input(j+4); %if isempty(alpha), alpha=0.1;

20 %mom=input(' momentum constant (between 0 and 1, default 0.8) mom = '); mom= input(j+5); %if isempty(mom), mom=0.8; % termination criteria % A. Terminate when the max. # of epochs to run is reached. %nepoch=input('maximum number of epochs to run, nepoch = '); nepoch= input(j+6); %disp(['# training samples = ' int2str(kr)]); %K = input(['epoch size (default = ' int2str(min(64,kr)) ', <= ' int2str(kr) ') = ']); K= input(j+7); %if isempty(k), K=min(64,Kr); %disp(['total # of training samples applied = ' int2str(nepoch*k)]); % B. Check the tuning set testing result periodically. If the tuning set testing % results is reducing, save the weights. When the tuning set testing results % start increasing, stop training, and use the previously saved weights. %disp('============================================================='); % nck=input(['# of epochs between convergence check (> '... % int2str(ceil(kr/k)) '): ']); nck= input(j+8); % disp(' '); % disp('if testing on tuning set meets no improvement for n0'); %maxstall=input('iterations, stop training! Enter n0 = '); maxstall= input(j+9); nstall=0; % initilize # of no improvement count. when nstall > maxstall, quit. if classreg==0, bstrate=0; % initialize classification rate on tuning set to 0 elseif classreg==1, bstss=1; % intialize tuning set error to maximum ssthresh=0.001; % initialize thres % training status monitoring E=zeros(1,nepoch); % record training error ndisp=5; % disp(' '); % disp(['the training error is plotted every ' int2str(ndisp) ' iterations']); % disp('enter <Return> to use default value. ') % chos1=input('enter a positive integer to set to a new value: '); chos1= input(j+10); if isempty(chos1), ndisp=5;

21 elseif chos1>0, ndisp=chos1; else ndisp=input('you must enter a positive integer, try again: '); % intialization for the bp iterations t = 1; % initialize epoch counter ptr=1; % initialize pointer for re-sampling the training file not_converged = 1; % not yet converged % save all variables into a file so that user needs not reenter all of them save mlpconfig.mat