ISDS 4141 Sample Data Mining Work. Tool Used: SAS Enterprise Guide

Transcription

1 ISDS 4141 Sample Data Mining Work Taylor C. Veillon Tool Used: SAS Enterprise Guide You may have seen the movie, Moneyball, about the Oakland A s baseball team and general manager, Billy Beane, who focused on an analytical, evidence-based, approach to assembling a competitive baseball team, despite Oakland's disadvantaged revenue situation. Suppose you have the following data from the 2008 MLB season and want to determine an appropriate model for predicting the Number of Wins (Y) based upon the number of Runs, Hits, Walks, Errors, and Saves. Save the data at the end of this document in an Excel spreadsheet name BB2008 and BB2009, respectively. (This assignment has 50 points which will be converted to 100 pts) BB2008 data is your training data and BB2009 is your validation data be sure to use the appropriate data for the specified questions below. Use what you have learned this semester to build the best model by answering questions 1 8: 1. Complete descriptive statistics, including means, standard deviations, minimum, maximum, and number of observations for all variables. Also include distributional analyses and comment on the normality of each variable. Indicate if there are any outliers when looking at the univariate variables. (Don t delete any outliers at this point they may not be influential in the regression analysis). (Please report to 1 decimal) (4 pts) Descriptive Statistics Wins (Y) Runs (X1) Hits (X2) Walks (X3) Errors (X4) Saves (X5) Mean Std Dev Min Max Sample Size Evidence of Non-normality Yes Yes Yes Yes Yes Yes Outliers - Yes Yes Yes Yes Yes (For non-normality and outliers, place a check in the box if your answer is Yes; otherwise leave blank)

2 2. Which team had the most Wins? California Angels (100 wins) (1 pt) Which team had the least Wins? Washington (50 wins) (1 pt) 3. Run a baseline model using the 2008 data to predict Number of Wins (Y) using all predictors. a. What is the estimated regression equation of your best (baseline) model? (2 pts) Yˆ = (X1) (X2) (X3) (X4) (X5) b. VALIDATE your results using BB2009 data. Attach a spreadsheet (like the one I provided in class when I discussed validation RMSE) showing the actual Y, predicted Y, error, error-squared and validation RMSE. Your validation RMSE is (5 pts) RMSE = c. Which predictor seems most related to Number of Wins (Y)? (1 pt) Saves (Semi partial correlation = ), and I know the predictor is definitely significant because its p value is less than alpha indicating that there is a relationship between the amount of saves and the number of wins for a team. d. Which predictor seems least related to Number of Wins (Y)? (1 pt) Errors (Semi partial correlation = ). However, this predictor is still potentially significant because its p-value is less than alpha. 4. To go along with the regression analysis, review a scatter plot matrix for Number of Wins (Y) and each predictor (X) to ensure that the linearity

3 assumption is reasonable (you don t have to include that output with your homework). For any situation where the relationship between Number of Wins (Y) and a predictor is non-linear, transform the predictor, so that the proper form of the predictor is used. Try to find the best transformation (both X 2 and X, ln(x), log(x), exp(x), sqrt(x), or 1/X). (NOTE: Always transform variables where necessary before checking for outliers. Remember correcting the form of the relationship may fix potential problems with outliers). a. Which predictor(s) need(s) transforming? (2 pts) Saves, Runs, Hits Saves:

4 Runs: Adjusted r squared is Taylor C. Veillon

5 Hits: is adjusted r squared. Appears that equal variance may be an issue. b. For each variable(s) needing transforming, what transformation was BEST? (4 pt) Saves(X2, X), RunsInverse, HitsLN

6 Saves: Variable Transformation Type Adjusted R 2 Saves SavesInverse SavesLN SavesSqRt SavesX SavesEXP The two highest adjusted R 2 values are for Saves inverse and SavesX 2. Of those two transformations, SavesX 2 appears to improve the linearity of the data more which leads me to assume that SavesX 2 is the best transformation for this variable. The p value is less than alpha so predictor is significant. Adjusted r squared is SavesX2-

7 Errors: Variable Transformation Type Adjusted R 2 Errors ErrorsInverse ErrorsLN ErrorsSqRt ErrorsX Taylor C. Veillon I have decided to leave errors in its original form. The original distribution of the data looks the best visually even though its adjusted r squared is not the highest.

8 Runs: Variable Transformation Type Adjusted R 2 Runs RunsInverse RunsLN RunsSqRt RunsX Taylor C. Veillon For runs, the two transformations with the highest adjusted R 2 values are RunsLN and RunsInverse. Of those two transformation types, RunsInverse does a better job with the linearity/distribution of the data leading me to believe that RunsInverse would be the best choice of possible transformations for this predictor. Its p-value is less than alpha indicating that it is significant in predicting wins, and its adjusted R 2 is equal to RunsInverse-

9 Variable Transformation Type Adjusted R 2 Walks WalksLN Walks SqRt WalksInverse WalksX Of the transformation options for the variable walks, the two highest adjusted R 2 values are for Walks and WalksX 2. Although WalksX 2 has a higher adjusted R 2, its effect on the distribution and linearity of the graph is bad, so I chose to keep the walks predictor in its original format. The p value is also less than alpha indicating that the predictor is significant. Walks-

10 Variable Transformation Type Adjusted R 2 Hits HitsLN HitsSqRt HitsInverse HitsX The adjusted R 2 values for the transformation types are incredibly similar for this predictor. However, the transformation with the best effect on the distribution and linearity of this predictor is HitsLN. It s p value is also less than alpha indicating it is significant.

11 HitsLN- Taylor C. Veillon

12 5. Run an analysis using 2008 data ONLY (as training data) to determine if there are any influential observations. Do this analysis using all of the predictors (taking into account any transformations you made in 4b above). a. What are the cutoffs for the following values for determining which observations are influential: leverage (h ii ) (1 pt) rstudent (t i ) _-2 or +2_(1 pt) DFFITS i or (1 pt) Cook s D i (1 pt) b. Based upon your cutoffs, If any, which observations have high leverage? None (1 pt) Why? No observations have an x value greater than the cutoff of (1 pt)

13 If any, which observations are discrepant (outliers)? (1 pt) Observation 14 (rstudent = ) Why? (1 pt) Observation 14 has an rstudent score beyond the cutoff of If any, which observations are influential? (1 pt) No observations are influential based on leverage and discrepancy alone. However, observation 14 does appear to have influence as well as observation 3 (once observation 14 is deleted). Why? (1 pt) Observation 14 has a DFFITS score of which is far beyond the cutoff of The Cooks D value is which is beyond the cutoff. Observation 14 also affects the slopes of all of the predictors slopes in my current prediction equation (runsinverse, HitsLN, Walks, Errors, Saves, SavesX2 and even the intercept). It appears to be affecting SavesX2 the most. Observation 20 was also a suspect for potential influence. It does have a DFFITS score of which is only slightly beyond the cutoff and is technically influencing the slope of LN(Hits). Cooks D for 20 is However, compared to observation 14, it does not appear to be influencing the prediction greatly; its leverage is also very low.

14 c. Assess the situation to see if you are justified in eliminating influential observations. Will you eliminate any influential observations? Yes or No (1 pt) Yes. Observation 14 and Observation 3. Why or why not? (1 pt) After eliminating observation 14, the adjusted r squared increased from.9094 to.9226.

15 Also using DFBETAS, the deletion statistic, the following calculations were made: DFBETAS cutoff is -0.4 to +0.4 Observation 14: DFBETASintercept = DFBETASRunsInverse = DFBETASHitsLN = DFBETASWalks = DFBETASErrors= DFBETASSaves= DFBETASSavesX2= Each of these values is beyond the cutoff zone further justifying my reasoning for deleting observation 14. Justification for observation 3 s deletion: (The influence of observation 3 became apparent after observation 14 was deleted.)

16 With outliers: Taylor C. Veillon

17 Without outliers: Taylor C. Veillon 6. At this point, you have decided whether or not to transformed your data and whether or not to delete influential observations. With those decisions made, conduct an all-possible-subset analysis and determine the best model (use 2008 data as TRAINING data)

18

19 a. Which predictors are included in your BEST model? (1 pt) runs-inverse (X1), hitsln (X2), saves (X3), savesx2 (X4) b. How did you arrive at that decision? (1 pt) This model has the highest adjusted r squared, the lowest root MSE, all of the predictors p-values are less than alpha indicating that they are significant in predicting wins. b. What is the estimated regression equation of your best model? (2 pts) Y hat = (X1) (X2) (X3) (X4) d. VALIDATE your results using BB2009 data. Attach a spreadsheet (like the one I provided in class when I discussed validation RMSE) showing the actual Y, predicted Y, error, error-squared and validation RMSE. Your validation RMSE is (5 pts) 7. Which regression model is BEST? The one you found in 3a or 6c? (2 pts) 3A Why did you choose the model you did? (2 pts) In my case, it appears the best model I picked in 6c could perhaps be over-fitted and therefore does not do a great job predicting random samples of the population. Therefore, in following the criteria for picking the best model based on validation, I would have to choose the original model in 3A because its validation RMSE is lower.

20 8. Use your best model provided in #7 above to explain to a coach what to do to increase the number of Wins? (4 pts) Yˆ = (X1) (X2) (X3) (X4) (X5) Based on the original model, I would explain that to increase the number of wins the coach should focus on two variables: number of runs and number of saves. Both have the two highest semi partial correlations indicating that they can explain the variance in wins the most of the five predictors. He should focus on increasing the team s number of runs by 5.574% and increase the number of saves by %. NOTE: The person with the BEST model will receive 4 extra credit points toward the total number of points for the semester.

21 2008 Team Runs Hits Walks Errors Saves Wins Arizona Atlanta Baltimore Boston Chicago Cubs Chicago White Sox Cincinnati Cleveland Colorado Detroit Florida Houston Kansas City California Angels Los Angeles Dodgers Milwaukee Minnesota New York Mets New York Yankees Oakland Philadelphia Pittsburgh San Diego Seattle San Francisco St. Louis Tampa Bay Texas Toronto Washington Taylor C. Veillon

22 2009 TEAMS Wins Runs Hits Walks Errors Saves Arizona Atlanta Baltimore Boston Chicago Cubs Chicago White Sox Cincinnati Cleveland Colorado Detroit Florida Houston Kansas City California Angels Los Angeles Dodgers Milwaukee Minnesota New York Mets New York Yankees Oakland Philadelphia Pittsburgh San Diego Seattle San Francisco St. Louis Tampa Bay Texas Toronto Washington