Complex Sampling Designs in Large-Scale Education Surveys: A Two-Level Sample Distribution Approach

Large-scale education data are collected via complex sampling designs that incorporate clustering and unequal probability of selection. Multilevel models are often utilized to account for clustering effects. The probability weighted approach (PWA) has been frequently used to deal with the unequal probability of selection. In this study, we examine the performance of an intuitive, easy to implement approach named the sample distribution approach (SDA) that utilizes Markov Chain Monte Carlo (MCMC) methods and Bayesian inference. Our simulation design focused on clustering effects, represented by the Intraclass Correlation (ICC) and on the sample size of the cluster. We analyzed a large-scale educational assessment dataset (Early Childhood Longitudinal Study - Kindergarten 2011) to compute estimates for the simulation. Findings reveal that the SDA overall generated reliable posterior distributions of parameters and had small error variances. In addition, although design informativeness is important, the ICC and cluster sample size factors had little impact on the performance of this model-based approach.