Methodological Features and Effect Sizes

In Recent Educational Research News...

A new article of interest was recently published in the Educational Researcher, which is arguably one of the most credible journals in educational research and is an official publication of the American Educational Research Association. So basically, we trust the stuff that's in here. In this article by Alan C. K. Cheung and Robert E. Slavin, titled "How methodological features affect effect sizes in education," the authors examined the effect sizes of 645 studies and looked to see whether there were differences between the different studies in terms of methodological features. If you've taken a basic methods course or an intro level statistics class, you will (hopefully) have learned that effect sizes are a way that researchers measure the practical significance of results, usually looking at differences in outcomes between groups, often receiving different "treatments" or "interventions."

So what did the authors find? It's more complex than this but, in a nutshell, they found that:

• Smaller studies (with smaller sample sizes) had twice the effect size compared to larger studies
• Published studies had higher effect sizes when compared to unpublished studies (like dissertations)
• Quasi-experimental studies had higher effect sizes than randomized studies
• Studies using researcher developed measures had higher effect sizes than those using independent, standardized measures.

What Cheung and Slavin argue with their findings is that we can't simply look at effect sizes to the exclusion of the methodological features of that study. They state, "...it is clear that researchers as well as policy makers need to take into account research design, sample size, measures and type of publications before comparing sizes from program evaluations" (Cheung & Slavin, 2016, p. 290). In some cases, certain studies overstate effect sizes and thus, overstate program impacts. Check out the study for some specific recommendations advanced by the authors.