Using Nonresponse Follow-up Recruitment to Help Build a Probability- Based Research Panel

Transcription

1 Using Nonresponse Follow-up Recruitment to Help Build a Probability- Based Research Panel N. Ganesh, Michael Dennis, and Paul J. Lavrakas May 13, 2016

2 Outline AmeriSpeak recruitment methodology Comparison of initially recruited panelists vs nonresponse follow-up (NRFU) panelists Sample distribution (fielded cases) Interview distribution (completed cases) Survey completion rate Cumulative response rate Design effect Substantive survey estimates 2

3 What is AmeriSpeak? NORC s AmeriSpeak is a household, multi-client panel Households selected from NORC s National Probability Frame are invited to join AmeriSpeak by telephone or web 3

4 4 AmeriSpeak Recruitment Methodology

5 Exemplar Study Details General population study of adults 18+ years 5,518 AmeriSpeak panelists were sampled; 2,834 completes (survey completion rate 51.4%) 1,567 completes among initially recruited panelists 1,267 completes among non-response follow-up (NRFU) panelists AAPOR Cumulative Response Rate 18.5% 5

6 Sample Distribution (Unweighted) by Panelist Type & Age Comparison of sample distribution for study by initially recruited panelists and NRFU panelists Initially Recruited Panelist NRFU Panelist

7 Sample Distribution (Unweighted) by Panelist Type & Phone Status Initially Recruited Panelist NRFU Panelist

8 Sample Distribution (Unweighted) by Panelist Type & Race/Ethnicity 70 Sampling NRFU panelists brought in more 60 Non-Hispanic Blacks 50 Hispanics years 30 HS graduate or less 20 never married 10 employed 0 Non-Hisp White Non-Hisp Black Non-Hisp Other Hispanic Non-Hisp 2+ races renters cell-only households Initially Recruited Panelist NRFU Panelist household size > 2 8

9 Interview Distribution* by Panelist Type & Age Comparison of interview distribution for study by initially recruited panelists and ALL panelists Initially Recruited Panelist ALL Panelist 15 CPS Benchmark 10 NRFU Reduces Bias by 38% *Weighted to account for probability of selection and NRFU 9

10 Interview Distribution* by Panelist Type & Race/Ethnicity Initially Recruited Panelist ALL Panelist CPS Benchmark NRFU Reduces Bias by 39% 10 0 White, non- Hispanic Black, non- Hispanic Hispanic All Other, non-hispanic *Weighted to account for probability of selection and NRFU 10

11 Interview Distribution* by Panelist Type & Housing Tenure Initially Recruited Panelist ALL Panelist CPS Benchmark NRFU Reduces Bias by 40+% 10 0 Owned Rented *Weighted to account for probability of selection and NRFU 11

12 Survey Completion Rate (Unweighted) by Panelist Type & Age Comparison of survey completion rate by initially recruited panelist and NRFU panelist Initially Recruited Panelist NRFU Panelist NRFU panelists have lower survey completion rates for the same demographic group 12

13 Cumulative Response Rate AAPOR Cumulative Response Rate (CRR) = (Weighted Panel Recruitment Rate) * (Weighted Panel Retention Rate) * (Survey Completion Rate) Compared CRR for this study with a hypothetical CRR for a study if we only had a panel of initial recruits Panel of Initial Recruits AmeriSpeak Panel Panel weighted recruitment rate 6.7% 36.9% Panel weighted retention rate 96.3% 97.7% Survey completion rate 58.3% 51.4% Cumulative response rate 3.8% 18.5% 13

14 Design Effect Re-weighted the survey by only considering completes among initially recruited panelists as being a complete. This replicates a panel of only initially recruited panelists (and no NRFU panelists). Re-weighted sample had a design effect of 1.9 while original set of completes (that includes the completes among NRFU panelists) had a design effect of

15 Substantive Survey Estimates Compared to initially recruited panelists, NRFU panelists are: LESS interested in current events, issues such as climate change, energy LESS likely to read a print newspaper MORE likely to read the news online, use Facebook, YouTube, obtain weather info online, pay bills online MORE likely to attend church, sports event, watch a movie in a theatre, eat at a fast food restaurant 15

16 How interested are you in current news events? Initially Recruited Panelist NRFU Panelist Combined 20.0 Initial Reweighted Very interested Moderately interested Not at all interested 16

17 Does the Earth go around the Sun, or does the Sun go around the Earth? Initially Recruited Panelist NRFU Panelist Combined Initial Reweighted Earth goes around Sun Sun goes around Earth Not sure 17

18 Summary Sample distribution: NRFU panelists are MORE likely to be younger, minority, and have high school degree or less Interview distribution: NRFU potentially reduces bias by ~40% Survey completion rate: NRFU panelists are less likely to complete surveys Cumulative response rate: NRFU has a 5x factor in improving the cumulative response rate Design effect: Relative to a panel without NRFU, NRFU does not adversely affect the design effect Substantive survey estimates: NRFU panelists make the findings of any AmeriSpeak study more accurate 18

19 N. Ganesh Thank You!