Interviewer Variance and its Effects n Estimates and Analyses Geert Lsveldt, Celine Wuyts, Ken Beullens Centre fr Scilgical Research
Outline Intrductin Tw appraches t evaluate interviewer effects The basic mdels A tw and a three level randm intercept mdel Presentatin f tw Cases Evaluatin f interviewer effects n the measurement f alchl cnsumptin in Eurpean cuntries Evaluatin f interviewer effects n attitudinal variables Discussin
Intrductin Evaluatin f interviewer effects is part f survey data quality assessment Tw appraches Interactin analysis f questin-answer sequences Deviances f the straightfrward interactin Interviewer behavir and related measurement errrs Interviewer variance analysis Dep Var= substantive variable; Indep Var= Interviewers Intra class crrelatin: prprtin f explained variance Only ne type f interviewer effect: differences between interviewers in the systematic effects f each interviewer n the answers
basic mdels Tw level hierarchical data structure: Respndents are nested within interviewers Tw level randm intercept (null) mdel Y ij Yij 0 j 0 j 00 00 0 j ij 0 j ij intercept fr interviewer j residual errr term fr respndent i; variance an interviewer-specific part f the intercept ;variance Interviewer effects= intra class crrelatin cefficient: int 2 u 2 u 2 e
Interviewer effects= intra class crrelatin cefficient: int 2 u Expressin f the hmgeneity f the respnses btained by the same interviewer Interviewers create additinal clustering in the data Impact n the interviewer design effect: deff r deff int ' int 2 u Variance inflatin due t interviewer effects 2 e 1 ( m 1) Var( y) Var( y) clu sint srs int
Three level hierarchical data structure: Repeated measurements at the lwest level nested within respndents and respndents are nested within interviewers Three level randm intercept (null) mdel With ; unique part f the intercept at the interviewer level the respndent level and the errr at the measurement level Intra class crrelatin cefficient at respndent and interviewer level:
Cases 1: Evaluatin f interviewer effects n the measurement f alchl cnsumptin in Eurpean cuntries Survey are used t cllect data abut alchl cnsumptin Data: Seventh rund f Eurpean Scial Survey (2014) Questins abut frequency and amunt f cnsumptin Cmbinatin f standardizatin and lcal adaptatin Difficult cncepts: typical frequency and standard drinks Lng recall perid: last twelve mnths Sensitive tpic Difficult questins abut a sensitive tpic risk fr interviewer effects
Variables One year abstinence: 1= n alchl cnsumptin during the last 12 mnths Frequent drinking: 1 = nn-abstaining respndents answering daily r Several times a week t the frequency f alchl cnsumptin (last 12 mnths) Amunt f alchl cnsumed n weekdays The amunt f beverage-quantity cmbinatins (e.g. glass f wine) during the last weekday respndents were drinking Unit: gram alchl Fr nn-abstaining respndents Amunt f alchl cnsumed n a weekend day
ICC s fr the alchl variables Cuntry One-year abstinence Frequent drinking Amunt cnsumed weekday Amunt cnsumed weekend AT 0.16 0.08 0.16 0.15 BE 0.11 0.02 0.02 0.02 CH 0.12 0.01 0.08 0.10 CZ 0.17 0.28 0.22 0.24 DE 0.08 0.02 0.03 0.09 DK 0.14 0.00 0.04 0.03 EE 0.20 0.11 0.11 0.22 ES 0.09 0.01 0.08 0.10 FI 0.00 0.03 0.01 0.00 FR 0.22 0.06 0.06 0.13 GB 0.17 0.09 0.07 0.07 HU 0.24 0.32 0.19 0.14 IE 0.11 0.07 0.18 0.10 IL 0.46 0.24 0.14 0.24 LT 0.15 0.30 0.19 0.21 NL 0.15 0.05 0.06 0.05 NO 0.06 0.04 0.08 0.10 PL 0.17 0.05 0.13 0.13 PT 0.09 0.12 0.14 0.14 SE 0.04 0.02 0.10 0.09 SI 0.22 0.05 0.09 0.05
The impact f interviewer effects n the evaluatin f the effect f gender n alchl cnsumptin Effect f gender n alchl cnsumptin The fixed effect mdel In all cuntries: wmen s alchl cnsumptin is substantially lwer in bth frequency and quantity A randm intercept mdel: Interviewer effects are taken int accunt Gender effects d nt change in directin and significant status
Estimated standard errrs fr the effect f gender in the fixed (blue) and randm intercept mdel (green) One-year abstinence FE and RI standard errr Frequent drinking FE and RI standard errr
The impact f interviewer effects n the evaluatin f the effect f educatin n alchl cnsumptin Binary variable: with and withut a degree f higher educatin The fixed effect mdel Educatinal level is differently related t the different dimensins f alchl cnsumptin In all cuntries: higher educatinal level: less likely t have abstained frm alchl In mst cuntries: higher educatinal level: mre likely t drink several times a week but smaller amunts A randm intercept mdel: A few f the bserved effects change in statistical significance status
Cases 2 : Evaluatin f interviewer effects n attitudinal variables Data ESS rund 7 Fur list f items (11 pint scales) abut different substantive tpics: A1: Trust in plitical institutins (7 items) A2: Satisfactin with (6 items, e.g. present state f ecnmy) A3: Imprtance f qualificatins fr freigners (6 items) A4: Apprecatin fr the plitical system (6 items)
Mean ICC fr fur list f attitudinal items A1 A2 A3 A4 Austria 0.21 0.18 0.15 0.14 Belgium 0.04 0.04 0.05 0.04 Switzerland 0.10 0.02 0.02 0.03 Czech Republic 0.20 0.26 0.25 0.12 Germany 0.07 0.07 0.08 0.07 Denmark 0.05 0.03 0.02 0.03 Estnia 0.11 0.09 0.12 0.11 Spain 0.11 0.05 0.07 0.06 Finland 0.03 0.01 0.01 0.03 France 0.02 0.03 0.03 0.02 United Kingdm 0.04 0.03 0.03 0.02 Hungary 0.29 0.35 0.33 0.32 Ireland 0.17 0.11 0.12 0.11 Israel 0.26 0.09 0.10 0.12 Lithuania 0.37 0.36 0.34 0.29 the Netherlands 0.04 0.02 0.02 0.03 Nrway 0.04 0.02 0.01 0.04 Pland 0.14 0.06 0.06 0.09 Prtugal 0.09 0.07 0.09 0.06 Sweden 0.04 0.01 0.02 0.03 Slvenia 0.04 0.03 0.03 0.06
Evaluatin f interviewer effects n (nn)-differentiatin Assumptin: respnse differentiatin n 11 pint scales Respnse style f nn-differentiatin r straightlining Can interviewers influence this respnse style and hw can we disentangle interviewer effects and respndent effects? Measurement f nn-differentiatin: Average deviatin between the current answer f a respndent as cmpared t the answer n the questin that precisely precedes it E.g. 6,7,8,6,6,6,4; deviatins are: NA,1,1,2,0,0,2 average deviatins 6/6=1
Hw can we separately evaluate interviewer effects and respndent effects? Repeated measurement mdel Three repeated measurements f nn-differentiatin Lists: A1, A2 and A3 Three level mdel: repeated measurement are nested within respndents and respndents within interviewers Variance cmpnent fr interviewers and respndents prprtin f explained variance by respndents (ICC_Resp) and by interviewers (ICC_int)
ICC-resp and ICC_int fr nn-differentiatin A1 A2 A3 A4 ICC.resp ICC.int Austria 0.21 0.18 0.15 0.14 0.05 0.08 Belgium 0.04 0.04 0.05 0.04 0.19 0.03 Switzerland 0.10 0.02 0.02 0.03 0.11 0.03 Czech Republic 0.20 0.26 0.25 0.12 0.07 0.14 Germany 0.07 0.07 0.08 0.07 0.15 0.03 Denmark 0.05 0.03 0.02 0.03 0.21 0.02 Estnia 0.11 0.09 0.12 0.11 0.11 0.05 Spain 0.11 0.05 0.07 0.06 0.14 0.03 Finland 0.03 0.01 0.01 0.03 0.21 0.00 France 0.02 0.03 0.03 0.02 0.20 0.01 United Kingdm 0.04 0.03 0.03 0.02 0.14 0.01 Hungary 0.29 0.35 0.33 0.32 0.01 0.17 Ireland 0.17 0.11 0.12 0.11 0.10 0.12 Israel 0.26 0.09 0.10 0.12 0.03 0.10 Lithuania 0.37 0.36 0.34 0.29 0.00 0.20 the Netherlands 0.04 0.02 0.02 0.03 0.20 0.01 Nrway 0.04 0.02 0.01 0.04 0.10 0.02 Pland 0.14 0.06 0.06 0.09 0.05 0.07 Prtugal 0.09 0.07 0.09 0.06 0.10 0.08 Sweden 0.04 0.01 0.02 0.03 0.15 0.02 Slvenia 0.04 0.03 0.03 0.06 0.11 0.01
Evaluatin f interviewers impact n nn-differentiatin n the crrelatin between attitudinal variables Interviewer effects n nn-differentiatin f respnses Nn differentiated answers that remain at the same side f the mean f the items will frce higher crrelatins Questin: What s the impact f the interviewer tendency t btain mre r less differentiated answers n the crrelatin between items f anther list Expectatin: Higher crrelatins fr interviewers with respndents that differentiate less
Analysis: Mean scre f nn-differentiatin fr each interviewer based n the three sets f items (A1,A2,and A3) Quintiles f the distributin f mean scres 5 grup f interviewers: Grup 1: 20% f interviewers with respndents with the lwest mean scre fr nn differentiatin Grup 5: 20% f interviewers with respndents with the highest mean scre fr nn differentiatin Average crrelatin between any tw items f 6 items f A4 fr the five grups f interviewers
Mean crrelatins between 6 items fr 5 grups f interviewers
Discussin High intra class crrelatins fr behaviral and attitudinal questins Large differences between cuntries Part f data quality assessment in a crss natinal surveys Detectin f gd and bad practices f interviewer training and fieldwrk mnitring Results underline the imprtance f training and fieldwrk mnitring
Discussin Interviewer effects have an influence n: effect parameters and standard errrs influence n substantial cnclusins Respnse style: (nn)-differentiatin nt nly a matter f respnse behavir Crrelatins between items results f statistical prcedures using crrelatin and cvariance matrices T reduce nn-differentiatin and the interviewer s impact: Scatter the items f ne cnstruct ver the questinnaire?
Thank yu.