Keynote speaker
Ronny Scherer
University of Oslo, Centre for Educational Measurement at the University of Oslo (CEMO) and Centre for Research on Equality in Education (CREATE)
Two Pillars, One Framework: How Meta-Analyses and ILSAs Can—and Should—Work Together
Meta-analyses and international large-scale assessments (ILSAs) are two of educational research’s powerful tools for generating policy-relevant evidence—yet they have largely developed in parallel. This keynote argues that treating these sources separately is a missed opportunity and presents a framework for bridging them.
Drawing on individual-participant and aggregated data meta-analysis, two complementary approaches are described: synthesizing effect sizes directly from ILSA data across countries and cycles and integrating ILSA-derived effects into broader meta-analyses alongside non-ILSA studies. Both approaches address well-known weaknesses of conventional meta-analyses—small samples, insufficient psychometric quality, and cultural selection bias—while leveraging ILSAs’ strengths in representativeness and measurement rigor.
The framework is illustrated through empirical examples examining effect sizes across countries, cycles, and subgroups. These examples reveal how integrated analyses can uncover meaningful divergences between ILSA and non-ILSA data sources in overall effects, heterogeneity, and moderator patterns—differences that remain invisible when the two evidence streams are analyzed in isolation.
The keynote closes with broader questions for the field: When should ILSAs be included in meta-analyses? What do discrepancies between evidence sources reveal? And what does integration mean for cumulative evidence in educational assessment?
Ronny Scherer
University of Oslo, Centre for Educational Measurement at the University of Oslo (CEMO) and Centre for Research on Equality in Education (CREATE)
Two Pillars, One Framework: How Meta-Analyses and ILSAs Can—and Should—Work Together
Meta-analyses and international large-scale assessments (ILSAs) are two of educational research’s powerful tools for generating policy-relevant evidence—yet they have largely developed in parallel. This keynote argues that treating these sources separately is a missed opportunity and presents a framework for bridging them.
Drawing on individual-participant and aggregated data meta-analysis, two complementary approaches are described: synthesizing effect sizes directly from ILSA data across countries and cycles and integrating ILSA-derived effects into broader meta-analyses alongside non-ILSA studies. Both approaches address well-known weaknesses of conventional meta-analyses—small samples, insufficient psychometric quality, and cultural selection bias—while leveraging ILSAs’ strengths in representativeness and measurement rigor.
The framework is illustrated through empirical examples examining effect sizes across countries, cycles, and subgroups. These examples reveal how integrated analyses can uncover meaningful divergences between ILSA and non-ILSA data sources in overall effects, heterogeneity, and moderator patterns—differences that remain invisible when the two evidence streams are analyzed in isolation.
The keynote closes with broader questions for the field: When should ILSAs be included in meta-analyses? What do discrepancies between evidence sources reveal? And what does integration mean for cumulative evidence in educational assessment?