Guidelines for Applying Multilevel Modeling to the NSCAW Data

Guidelines for Applying Multilevel Modeling to the NSCAW Data by Sharon Christ, Paul Biemer and Christopher Wiesen Odum Institute for Research in Social Science May 2007 Summary. This document is intended to provide (1) the current best approach to the use of survey weights with multilevel models, (2) a brief description of appropriate weights to use, and (3) general instructions for the proper application of NSCAW multilevel weights in several software packages. The target audience for these guidelines is data analysts who are already familiar with multilevel modeling methodology but have little or no experience in dealing with weighted multilevel models. More detailed instructions for users who are not as familiar with multilevel modeling are forthcoming in the next version of the NSCAW Data Analysis Manual. We welcome feedback and questions on this document and the multilevel weights. Please send inquiries and suggestions to Sharon Christ (slchrist@email.unc.edu). I. Introduction The NSCAW data arise from a multistage sample design resulting in children nested within primary sampling units (PSUs). A PSU is usually a county but may be part of a large county or a combination of two or more small counties. Within each PSU, a sample of children is selected who may be interviewed up to five times. Thus, observations are nested within children, which are also nested within PSUs to create three levels of units. Most NSCAW analysts are interested in characteristics of agencies, not PSUs. In order to use to introduce the agency as a level in multilevel modeling, weights (i.e., inverse selection probabilities) must be computed at the agency level using the probabilities of selection assigned to the PSUs. One issue in this process is that the definition of an agency is somewhat ambiguous. Since many researchers would like to use the information obtain in the NSCAW Local Agency Directors Survey (LADS), we define an agency to be consistent with the definition used in that survey. The LADS defined the agency to be the administrative unit responsible for child welfare investigations and services rather than a unit defined by physical boundaries. This definition may be problematic for some investigations since the scope of an administrative unit can vary by state and/or locality. Statistical software developed for survey data analysis such as SUDAAN and STATA can accommodate clustering and will provide valid estimates and their standard errors. However, for studying the effects of county or agency characteristics on child outcomes, multilevel modeling (also known as hierarchical linear modeling, mixed modeling, and random effects modeling) may be preferred 1. SUDAAN and STATA cannot perform some types of multilevel modeling so a different software package such as MLWin, HLM, etc. must be used. As an example, an analyst may wish to fit multilevel models for two levels, for example, children nested within agencies or wave of interview nested within children. A three level model may also be fit to the NSCAW data with interview wave nested within children nested within 1 The next version of the NSCAW Statistical Analysis Manual will contain a fuller discussion of the differences between multilevel modeling with SUDAAN and STATA and that of special multilevel modeling software. 1

agency. For the two-level or three-level model, special weights must be used for valid inference as we shall discuss in this document. For dealing with nested data structures, multilevel modeling is an alternative to the approaches used by survey analysis software such as SUDAAN and STATA. Cluster sampling violates common statistical assumptions of homogenous error variance and independent observations. Survey analysis software packages like SUDAAN and STATA handle clustered data by essentially fitting a marginal or population average model. Instruction on how to analyze marginal models using the NSCAW data in these packages is outlined in the NSCAW Statistical Analysis Manual (Biemer & Christ, 2005). Marginal model analysis of complex data structures like NSCAW is well developed and clustering, stratification, and sampling weights are easily accommodated with these methods. Unfortunately, these software packages do not have the capability to fit multilevel models with random effects. Multilevel modeling software such as HLM, MLWIN, LISREL, GLLAMM, and MPLUS can handle more sophisticated and complex multilevel models. However, these packages have only recently added features that attempt to compensate for unequal probability sampling (i.e., weighting). In fact, the appropriate methods for treating unequal probability sampling at multiple levels used in this document are still being developed and should be considered experimental. The statistical literature has only recently provided guidance on applying sampling weights in multilevel models. The purpose of this report is to share this guidance with the NSCAW user community and to provide our recommendations for when and how to use multilevel modeling with the NSCAW data set. A more elaborate discussion of multilevel modeling using the NSCAW data is in preparation and will be provided in the next version of the NSCAW Data Analysis Manual. II. Using NSCAW Sampling Weights in Multilevel Models As noted in Biemer & Christ (2005), regardless of the model, the NSCAW data should be analyzed using sampling weights to minimize bias in parameter and standard error estimates. General child-level weights are provided on the NSCAW data files. Unfortunately, when multilevel analysis software is implemented, these child-level weights are not appropriate. Instead, a separate set of weights must be used which are not available on the data file, but can be acquired through the National Data Archive on Child Abuse and Neglect (NDACAN) for licensed restricted release users. Similarly, the weights that are available for multilevel modeling through NDACAN are not appropriate for traditional child-level modeling methods. Care must be taken to apply the appropriate weights for a given modeling approach. What follows is a description of the multilevel model weighting methods that are currently recommended by the statistical literature and implemented in several multilevel modeling software packages. As previously noted, the NSCAW sample can be viewed as a two-stage stratified sampling design for the purposes of multilevel modeling. The first sampling stage (the county) was selected with probability proportionate to size which means that the NSCAW sample 2

contains a disproportionate number of large CPS agencies. Failure to compensate for this disproportionality in the modeling at the agency level will bias the results toward large agencies. In addition, within each selected agency, children were selected with unequal probability sampling since foster children, infants, sexually abused children and open cases were oversampled. Therefore, an appropriate multilevel analysis of the NSCAW data which includes the agency as one level and the child within agency as a second level must include specially developed sampling weights for the agency and child. It should be emphasized that the child-level weights in this situation are not the standard NSCAW single-level weight which includes both the probability that the PSU was selected as well as the selection of the child within the PSU. Also, the agency-level weight in this analysis is not the same as a PSU-level weight since agencies, strictly speaking, are not synonymous with PSUs. Simply using the standard NSCAW single-level weight at the child level would produce biased parameter estimates at the other level(s) in a multilevel model. Likewise, for a three-level model where the agency corresponds to level-3, the child is level-2 and the NSCAW wave is level-1, separate, specially developed weights must be applied to each of the three levels and these multilevel weights are also currently not available on the NSCAW data file, but are included on the multilevel weight file. For the multilevel modeling scenarios described above, the appropriate weights are available upon request from the NDACAN archive at Cornell University (http://www.ndacan.cornell.edu) for all researchers who hold a current restricted release authorization. Simply request the Multilevel Weight File and the file will be sent to you. A. Description of Multilevel Weights This section describes how the multilevel weights on the Multilevel Weight File were created. It is not essential for analysts to understand this process in order to use the weights. However, we include this discussion for some who may be interested in the nature of multilevel weights and how they differ from the standard weights that are used in NSCAW single-level analysis. Others may skip to Section B. Sampling weights are required whenever informative sampling is used as it was in the NSCAW. To do otherwise may result in bias in parameter estimation (Pfeffermann, et al., 1998). Informative sampling occurs when the inclusion probabilities are related to an outcome variable after controlling for other covariates. Sampling may be informative at any level of a multilevel model. For example, in a two-level model inclusion probabilities at level-2 (say, the agency) may be informative if the agency characteristics used to select agencies and not included in the model are related to the child-level dependent variable. Likewise, inclusion probabilities at level-1 may be informative since the domain variables used to oversample children within agencies are related to many child outcomes and, for most analyses, these domain variables would not be included in the model. The basic requirement to compensate for the biasing effects of unequal inclusion probabilities, which may be informative at any level, is to employ sampling weights at each level 3

that are functions of the selection probabilities for that level (Skinner, 2005). Weights for a twolevel model in NSCAW with agency at level-2 and individual at level-1 are: Level-2 weight (AGYWGT): 1 w j = π j where π j is the probability of inclusion for agency j Level-1 weight: 1 wi j = π i j where π i j is the conditional probability of inclusion for child i within agency j, given that agency j is included in the sample Note that the above person-level weights are not adjusted for nonresponse. Indeed, the appropriate statistical methodology for adjusting for nonresponse in multilevel weighting is still being investigated. The current recommended approach for compensating for nonresponse at the person level is to form the person-level weight by dividing the standard NSCAW single-level weight (which has been fully adjusted for both nonresponse and frame noncoverage) by the agency level weight that was computed above. Thus, wij Level-1 weight (PERSONWT): wij = where w ij is the unconditional person-level weight w j that has been adjusted for nonresponse and frame noncovereage. Research has shown that scaling the level-1 weight decreases bias in random effects parameters and improve efficiency (Skinner, 2005, Pfefferman, et al., 1998). The recommended scaling proportionally adjusts the size of the level-1 weight so that the sum of those weights equals the level-1 sample size within each level-2 unit. For example, the sum of the level-1 weights in Agency A is equal to the number of children in the sample from Agency A. This scaling is automatically done in the multilevel software packages discussed in these guidelines. Level-1 weights for two-level analyses using variables at either wave 3 or 4 are created in a similar manner. These variables are PERSONWT3 and PERSONWT4. The same agency weight (AGYWT) is used for analysis at any of the three waves. To summarize, the appropriate weights for a two-level model with agency at level two are: Wave 1 Wave 3 Wave 4 Level-2 weight AGYWGT AGYWGT AGYWGT Level-1 weight PERSONWT PERSONWT3 PERSONWT4 For a three-level model where agency is level three, child is level two, and wave of interview (or time) is level one, AGYWGT and PERSONWT are the appropriate weights to use at levels three and two, respectively. However, now conditional weights need to be used at level one. This conditional weight is essentially the inverse of the probability of response (i.e., response propensity) at each wave. For wave 1, this conditional weight is set equal to one since, as noted above, a response propensity weight adjustment for wave 1 was already incorporated wij into the PERSONWT. For wave 3 the appropriate weight to use is TIME3WT = and the w 3ij 4

w weight for wave 4 is TIME4WT = w ij 4ij. It can be shown that these weights are essentially equivalent to the inverse of the probability of response at corresponding wave. In this way, the wave 1, 3 and 4 conditional weights incorporate adjustments for nonresponse. The TIMEWT needs to be constructed by the analyst. This is because the multilevel software programs accept rectangular datasets (sometimetimes referred to as person-year datasets) where there is an observation for each observed wave for each respondent; each respondent may have between one and three records in the dataset. Then the level 1 weight is constructed as: TIMEWT = TIME1WT for an observation at wave 1, = TIME3WT for an observation at wave 3, and = TIME4WT for an observation at wave 4. Formatting of the data set for a three-level analysis is described in the next section. To summarize, the appropriate weights for a three-level model is: Level-3 weight (AGYWGT) Level-2 weight (PERSONWT) Level-1 weight (TIMEWT) B. Two-level Models with Child at Level-2 and Time at Level-1 Many researchers wish to analyze change over time using the NSCAW. This is a specific type of two-level multilevel model where children are level-2 observations and time (wave of collection) is level-1. These types of models may be analyzed in a marginal model, which allows for testing most of the common hypotheses of change. These hypotheses and instructions of fitting these models are provided in the NSCAW Statistical User s Manual, Section 5 (Biemer & Christ, 2005). We recommend using this approach rather than a multilevel modeling approach since most multilevel modeling packages do not allow you to estimate this two-level model and correct for clustering at the PSU level. An alternative approach to fitting these two-level models that does accommodates agency clustering as well as stratification is the structural equation modeling approach to analysis of change (Bollen & Curran, 2006). LISREL, Mplus, and GLLAMM software are capable of fitting these models. Contact Sharon Christ (slchrist@email.unc.edu) for more information. 5

C. Multilevel Weight File The multilevel weight file (multiwgts) contains the following variables: Variable Description NSCAWID Child identifier NSCAWPSU PSU identifier AGENCYID Administrative unit (agency) identifier. This is not the same as NSCAWAGN AGYWGT Administrative unit (agency) weight PERSONWT Conditional child within agency weight at wave 1 PERSONWT3 Conditional child within agency weight at wave 3 PERSONWT4 Conditional child within agency weight at wave 4 TIME1WT 1 = Conditional response at wave 1 given child inclusion weight TIME3WT Conditional response at wave 3 given child inclusion weight TIME4WT Conditional response at wave 4 given child inclusion weight This weight file should be merged with the CPS data file using the NSCAWID variable. Once the files are merged, the data are ready for cross-sectional, two-level models with child nested within agency. However, for three-level model analysis, software may require observations to be child-by-time or wave-level data. In this case, the data need to be reformatted where the weights for each of the waves are combined into a single TIMEWT. C. Multilevel Modeling Software and Weights To date, software that handles multilevel modeling has varied greatly in the manner in which weights are handled (Chantala, 2006). The method of weighting described above currently may be implemented in several software packages including LISREL 8.8, MPLUS 4.2, HLM 6.04, and GLLAMM (a STATA program). Note that prior versions of some of these software packages applied multilevel weights in a less desirable way. Other multilevel packages such as MLWin and SuperMix are expected to eventually come in line with the same weighting method resulting in consistent weighted estimation across packages. 6

II. Software Instructions for Two-Level Models using NSCAW LISREL A. Example for Two-level Models with Agency at Level-2 and Child at Level-1 LISREL allows the user to specify a sampling weight at each level and automatically applies scaling of the weights. Multilevel models may be specified using LISREL windows interface. Weight variables are assigned under the multilevel identification variables tab: The user may identify up to three weights for each of three levels. For a two level model, the agency id variable (AGENCYID) is specified as the Level-2 ID Variable. The agency weight (AGYWGT) is specified as the Level-2 Weight and the child within agency weight (PERSONWT) is specified as the Level-1 Weight (for wave 1 analysis). 7

Multilevel models may also be specified using syntax in LISREL. Weights at various levels are assigned using the weight1, weight2, and weight3 commands. The following is an example of a random intercepts model with NSCAW two-level weights applied. OPTIONS OLS=YES CONVERGE=0.001000 MAXITER=10 OUTPUT=STANDARD; TITLE=Random Intercepts Model with Two-level Weights; SY='C:\data.psf'; ID2=AGENCYID; WEIGHT2=AGYWGT; WEIGHT1=PERSONWT; RESPONSE=pbc_tot; FIXED=intcept; RANDOM1=intcept; RANDOM2=intcept; 8

HLM HLM allows the user to specify a sampling weight at each level and automatically applies scaling of the weights. In HLM, a separate data file for each level is required. When creating these file, be sure to include the AGYWGT variable in the level-2, agency file and the PERSONWT variable in the level-1, child file. Multilevel models are specified using a windows interface. HLM allows specification of up to three weights for each of three levels. Weight variables are assigned under the other settings estimation setting tab: The Estimation Setting window will come up, then click on Weighting. For a two level model, the agency weight (AGYWGT) is specified as the Level-2 Weight and the conditional child weight (PERSONWT) is specified as the Level-1 Weight (for a wave 1 analysis). 9

Multilevel models may also be specified using an interactive and batch mode in HLM. Weights at each level are assigned using the LEVEL1WEIGHT: and LEVEL2WEIGHT: commands. For the NSCAW model, the weight variables are specified as follows: LEVEL2WEIGHT: AGYWGT; LEVEL1WEIGHT: PERSONWT; 10

MPLUS: MPLUS allows the user to specify a sampling weight at each level and to select among several weight scaling options. In MPLUS, multilevel models are specified using syntax only. For a two level model, the agency id variable (AGENCYID) is specified as the cluster variable. Weight variables are assigned using the WEIGHT and BWEIGHT commands, where weight specifies the within-level or level one weight and bweight specifies the between-level or level-2 weight. The default scaling in Mplus should be used so That the WTSCALE and BWTSCALE commands do not need to be specified. The following is an example of a random intercepts model with NSCAW two-level weights applied. title: Random intercepts model with two level weights data: file is data.dat; variable: names are AGENCYWT NSCAWAGN NSCAWID PERSONWT PBC_TOT; usevariables are pbc_tot; missing are.; cluster is agencyid; weight is personwt; bweight is agencywt; analysis: type = meanstructure twolevel; 11

III. REFERENCES Biemer, P. P. and Christ, S. L. (2005). NSCAW Statistical Analysis Manual. Ithaca, NY: National Data Archive on Child Abuse and Neglect. Bollen, K. A. and Curran, P. J. (2006) Latent Curve Models: A Structural Equation Perspective. New York: Wiley. Chantala, K., Suchindran, C. (2006) Adjusting for Unequal Selection Probability in Multilevel Models: A Comparison of Software Packages. Proceedings of the American Statistical Association, Seattle, WA: American Statistical Association. Pfefferman, D., Skinner, C. J., Holmes D. J., Goldstein, H., and Rasbash, J., (1998). Weighting for Unequal Selection Probabilities in Multilevel Models. Journal of the royal Statistics Society, B, 60, 123-140. Skinner, C. J., (2005). The Use of Survey Weights in Multilevel Modelling. Presented at the Workshop on Latent Variable Models and Survey Data for Social Science Research. Montreal, Canada. May, 2005. 12