Announcements. % College graduate vs. % Hispanic in LA. % College educated vs. % Hispanic in LA. Problem Set 10 Due Wednesday.

Transcription

1 Announcements Announcements UNIT 7: MULTIPLE LINEAR REGRESSION LECTURE 1: INTRODUCTION TO MLR STATISTICS 101 Problem Set 10 Due Wednesday Nicole Dalzell June 15, 2015 Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Recap % College graduate vs. % Hispanic in LA What can you say about the relationship between of % college graduate and % Hispanic in a sample of 100 zip code areas in LA? Recap % College educated vs. % Hispanic in LA What can you say about the relationship between of % college graduate and % Hispanic in a sample of 100 zip code areas in LA? Education: College graduate 1.0 Race/Ethnicity: Hispanic % % College graduate 75% 50% 25% % Freeways No data 0.0 Freeways No data 0.0 0% 25% 50% 75% 100% % Hispanic Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1

2 Recap % College educated vs. % Hispanic in LA - linear model Recap % College educated vs. % Hispanic in LA - linear model Participation question Which of the below is the best interpretation of the slope? (Intercept) %Hispanic (a) A 1% increase in Hispanic residents in a zip code area in LA is associated with a 75% decrease in % of college grads. (b) A 1% increase in Hispanic residents in a zip code area in LA is associated with a 0.75% decrease in % of college grads. (c) An additional 1% of Hispanic residents decreases the % of college graduates in a zip code area in LA by 0.75%. (d) In zip code areas with no Hispanic residents, % of college graduates is expected to be 75%. Do these data provide convincing evidence that there is a statistically significant relationship between % Hispanic and % college graduates in zip code areas in LA? (Intercept) hispanic How reliable is this p-value if these zip code areas are not randomly selected? Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Recap Recap Inference for the slope for a SLR model (only one explanatory variable): Hypothesis test: T = b 1 null value SE b1 df = n 2 Dinosaur Weight SLR: Categorical Predictors What relationship do you see between the weight of dinosaurs and the type of dinosaur? Dinosaur Weight by Type Confidence interval: b 1 ± t df=n 2 SE b 1 The null value is often 0 since we are usually checking for any relationship between the explanatory and the response variable. The regression output gives b 1, SE b1, and two-tailed p-value for the t-test for the slope where the null value is 0. We rarely do inference on the intercept, so we ll be focusing on the estimates and inference for the slope. Weight (kg) 0e+00 4e+04 8e+04 Ornithischian Saurischian Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1

3 SLR: Categorical Predictors SLR: Categorical Predictors Dinosaur Weight Dinosaurs! What relationship do you see between the weight of dinosaurs and the type of dinosaur? (Intercept) dino$typesaurischian Weight = TypeSaurischian Type of dinosaur is a categorical variable with two levels: Ornithischian and Saurischian For Ornithischian dinosaurs: plug in 0 for TypeSaurischian For Saurischian dinosaurs: plug in 1 for TypeSaurischian Slope b 1 : We expect that Saurischian dinosaurs weighed on average 13,652 kilograms more than Ornithischian dinosaurs. Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Return to the scene of the crime Murder Rates by Country Last class, we used the poverty rate in a district to help predict the number of annual murders per million in that district. murders = poverty country-info/ stats/ Crime/ Murders/ Per-capita (Intercept) percpov e-06 Do we think that poverty rates are the only thing that influence the number of annual murders per million in a district? Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1

4 Data from the ACS Data from the ACS A random sample of 783 observations from the 2012 ACS. 1 income: Yearly income (wages and salaries) 2 employment: Employment status, not in labor force, unemployed, or employed 3 hrs work: Weekly hours worked 4 race: Race, White, Black, Asian, or other 5 age: Age 6 gender: gender, male or female 7 citizens: Whether respondent is a US citizen or not 8 time to work: Travel time to work 9 lang: Language spoken at home, English or other 10 married: Whether respondent is married or not 11 edu: Education level, hs or lower, college, or grad 12 disability: Whether respondent is disabled or not 13 birth qrtr: Quarter in which respondent is born, jan thru mar, apr thru jun, jul thru sep, or oct thru dec We have 1 response variable (income) and 12 potential explanatory variables. How do we fit a model and interpret the coefficients with so many predictors? What do we do with a mix of categorical and numerical explanatory variables? How do we deterimine which (if any) of them are important in our model? How do we determine if our model is any good? Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Everybody in the Pool Examples How would we interpret the coefficient for hrs work? How do we interpret all of this? Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1

5 In MLR everything is conditional on all other variables in the model Examples MLR (Multiple Linear Regression) How would we interpret the coefficient for hrs work? In MLR everything is conditional on all other variables in the model. All estimates in a MLR for a given variable are conditional on all other variables being in the model. Slope: Numerical x: All else held constant, for one unit increase in x i, y is expected to be higher / lower on average by b i units. Categorical x: All else held constant, the predicted difference in y for the baseline and given levels of x i is b i. Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 In MLR everything is conditional on all other variables in the model In MLR everything is conditional on all other variables in the model Examples How would we interpret the coefficient for genderfemale? Examples How would we interpret the coefficient for genderfemale? Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1

6 In MLR everything is conditional on all other variables in the model In MLR everything is conditional on all other variables in the model Categorical Predictors with multiple levels We have several categorical variables in this study. Some are binary (ie have only two levels) but others are not. employment: Employment status, not in labor force, unemployed, or employed race: Race, White, Black, Asian, or other gender: gender, male or female citizens: Whether respondent is a US citizen or not lang: Language spoken at home, English or other married: Whether respondent is married or not edu: Education level, hs or lower, college, or grad disability: Whether respondent is disabled or not birth qrtr: Quarter in which respondent is born, jan thru mar, apr thru jun, jul thru sep, or oct thru dec Birth Quarter Coefficients birth qrtr: Quarter in which respondent is born, jan thru mar, apr thru jun, jul thru sep, or oct thru dec How many coefficients do we see for birth qrtr? Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 In MLR everything is conditional on all other variables in the model Categorical predictors and slopes for (almost) each level Race Coefficients Categorical predictors and slopes for (almost) each level How many coefficients do we see for birth qrtr? When we are working with a categorical variable with k levels, we only see k 1 parameters being estimated in the model. This happens because one of the levels of the variable is consumed by the intercept of the model. This level of the categorical variable is called the baseline. So, what happened to the folks born in January through March? Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1

7 Categorical predictors and slopes for (almost) each level Categorical predictors and slopes for (almost) each level Gender: male / female (k = 2) Respondent gender:female Female 1 Male 0 Estimate Std. Error t value Pr(> t ) Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Birth Quarter: (k = 4) Baseline: Jan thru Mar Respondent birth qrt:apr thru jun birth qrtr:jul thru sep birth qrtr:oct thru dec 1, jan thru mar , apr thru jun , jul thru sep , oct thru dec Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Participation question Categorical predictors and slopes for (almost) each level All else held constant, how do incomes of those born January thru March compare to those born April thru June? All else held constant, those born Jan thru Mar make, on average, (a) $2, less (b) $2, more than those born Apr thru Jun. (c) $ less (d) $ more Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Prediction with MLR Return to the scene of the crime Predict the annual murders per million in a district with a poverty rate of 24%. murders = percpov (Intercept) percpov e-06 Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1

8 Prediction with MLR Prediction with MLR Weights of books Predicting for MLR weight (g) volume (cm 3 ) cover hc hc hc hc hc hc hc pb pb pb pb pb pb pb pb l w h Write down the linear model for book weight based on volume and cover type. (Intercept) volume cover:pb Interpret the coefficients for volume and cover:pb. Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Prediction with MLR Interpretation of the regression coefficients Prediction Prediction with MLR (Intercept) volume cover:pb Participation question Which of the following is the correct calculation for the predicted weight of a paperback book that is 600 cm 3? Slope of volume: All else held constant, for each 1 cm 3 increase in volume we would expect weight to increase on average by 0.72 grams. Slope of cover: All else held constant, the model predicts that paperback books weigh on average grams less than hardcover books. Intercept: Hardcover books with no volume are expected on average to weigh 198 grams. Obviously, the intercept does not make sense in context. It only serves to adjust the height of the line. Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 (Intercept) volume cover:pb (a) (b) (c) (d) Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1

9 Examining our Model How do we determine if the model is significant? We can now interpret the coefficients, but how do we know if the model is any good? To be more specific, how can we show that the model is significant? Inference for the model as a whole: F-test Degrees of Freedom: df 1 = p, df 2 = n k 1 H 0 : β 1 = β 2 = = β k = 0 H A : At least one of the β i 0 Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Model output Coefficients: (Intercept) hrs_work e-12 *** raceblack raceasian ** raceother age e-05 *** genderfemale e-06 *** citizenyes time_to_work langother marriedyes educollege *** edugrad < 2e-16 *** disabilityyes * birth_qrtrapr thru jun birth_qrtrjul thru sep birth_qrtroct thru dec Residual standard error: on 766 degrees of freedom (60 observations deleted due to missingness) Multiple R-squared: , Adjusted R-squared: F-statistic: on 16 and 766 DF, p-value: < 2.2e-16 Participation question True / False: The F test yielding a significant result means the model fits the data well. (a) True (b) False Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1

10 Significance also depends on what else is in the model Weights of books Model 1: (Intercept) hrs_work e-12 raceblack raceasian raceother age e-05 genderfemale e-06 citizenyes time_to_work langother marriedyes <---- educollege edugrad < 2e-16 disabilityyes birth_qrtrapr thru jun birth_qrtrjul thru sep birth_qrtroct thru dec Model 2: (Intercept) hrs_work e-15 raceblack raceasian e-06 raceother age e-05 genderfemale e-05 marriedyes <---- weight (g) volume (cm 3 ) cover hc hc hc hc hc hc hc pb pb pb pb pb pb pb pb l w h Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Weights of hard cover and paperback books Modeling weights of books using volume and cover type Can you identify a trend in the relationship between volume and weight of hardcover and paperback books? # load data library(daag) data(allbacks) weight (g) hardcover paperback # fit model book_mlr = lm(weight volume + cover, data = allbacks) summary(book_mlr) Coefficients: (Intercept) ** volume e-08 *** cover:pb *** volume (cm 3 ) Residual standard error: 78.2 on 12 degrees of freedom Multiple R-squared: , Adjusted R-squared: F-statistic: on 2 and 12 DF, p-value: 1.455e-07 Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1

11 Linear model Visualising the linear model (Intercept) volume cover:pb weight = volume cover : pb 1 For hardcover books: plug in 0 for cover weight = volume = volume 2 For paperback books: plug in 1 for cover weight = volume = volume weight (g) hardcover paperback volume (cm 3 ) Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Interpretation of the regression coefficients Prediction (Intercept) volume cover:pb Participation question Which of the following is the correct calculation for the predicted weight of a paperback book that is 600 cm 3? Slope of volume: All else held constant, for each 1 cm 3 increase in volume we would expect weight to increase on average by 0.72 grams. Slope of cover: All else held constant, the model predicts that paperback books weigh 184 grams lower than hardcover books, on average. Intercept: Hardcover books with no volume are expected on average to weigh 198 grams. Obviously, the intercept does not make sense in context. It only serves to adjust the height of the line. Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 (Intercept) volume cover:pb (a) (b) (c) (d) Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1

12 A note on interaction variables weight = volume cover : pb volume (cm 3 ) weight (g) hardcover paperback This model assumes that hardcover and paperback books have the same slope for the relationship between their volume and weight. If this isn t reasonable, then we would include an interaction variable in the model (beyond the scope of this course). Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Revisit: Modeling poverty poverty metro_res white hs_grad female_house Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Predicting poverty using % female householder # load data poverty = read.csv(" mc301/data/poverty.csv") # fit model pov_slr = lm(poverty female_house, data = poverty) summary(pov_slr) Linear model: (Intercept) female house % female householder % in poverty R = 0.53 R 2 = = 0.28 Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Another look at R 2 - from last time anova(pov_slr) ANOVA: Df Sum Sq Mean Sq F value Pr(>F) female house Residuals Total SS of y: SS Tot = (y ȳ) 2 = total variability SS of residuals: SS Err = e 2 i = unexplained variability SS of regression: SS Reg = SS Total SS Error explained variability = = R 2 = explained variability total variability = = 0.28 Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1

13 Predicting poverty using % female hh + % white pov_mlr = lm(poverty female_house + white, data = poverty) summary(pov_mlr) anova(pov_mlr) Linear model: (Intercept) female house white ANOVA: Df Sum Sq Mean Sq F value Pr(>F) female house white Residuals Total ( R 2 SSError adj = 1 n 1 ) SS Total n k 1 where n is the number of cases and k is the number of predictors (explanatory variables) in the model. R 2 = explained variability total variability = = 0.29 Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Application exercise: Calculate adjusted R 2 for the multiple linear regression model predicting % living in poverty from % female householders and % white. Remember n = 51, 50 states + DC. (a) 0.26 (b) 0.29 (c) 0.32 (d) 0.71 ANOVA: Df Sum Sq Mean Sq F value Pr(>F) female house white Residuals Total R 2 vs. adjusted R 2 R 2 Model 1 (poverty vs. female house) Model 2 (poverty vs. female house + white) 0.29 When any variable is added to the model R 2 increases. But if the added variable doesn t really provide any new information, or is completely unrelated, adjusted R 2 does not increase. Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1

14 - properties R 2 adj = 1 ( SSError SS Total n 1 n k 1 ) Because k is never negative, R 2 adj will always be smaller than R 2. R 2 adj applies a penalty for the number of predictors included in the model. Therefore, we choose models with higher R 2 adj over others. Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Participation question True or false: tells us the percentage of variability in the response variable explained by the model. (a) True (b) False Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Collinearity and parsimony We saw that adding the variable white to the model did not increase adjusted R 2, i.e. did not add any valuable information to the model. Why? poverty metro_res white hs_grad female_house Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1 Collinearity and parsimony Collinearity between explanatory variables (cont.) Two predictor variables are said to be collinear when they are correlated, and this collinearity (also called multicollinearity) complicates model estimation. Remember: Predictors are also called explanatory or independent variables, so they should be independent of each other. We don t like adding predictors that are associated with each other to the model, because often times the addition of such variable brings nothing to the table. Instead, we prefer the simplest best model, i.e. parsimonious model. In addition, addition of collinear variables can result in biased estimates of the slope parameters. While it s impossible to avoid collinearity from arising in observational data, experiments are usually designed to control for correlated predictors. Statistics 101 (Nicole Dalzell) U7 - L1: Multiple Linear Regression June 15, / 1