Person and Context as Predictors of Program Impact
The standardized nature of many academic interventions ensures consistency across schools. In the case of Reading Corps and Math Corps, that consistency helps schools be confident that they will see effects similar to those observed in large-scale evaluations.
Yet, like all educational interventions, we know the impact of Reading Corps and Math Corps varies across schools and tutors. Using our data management system—as well as supplemental datasets from our partners—our research activities under Person and Context as Predictors of Program Impact are designed to identify factors outside the program that might affect student learning. We are particularly interested in identifying patterns that can be leveraged to improve the way in which our programs meet the needs of students.
The Wayne Gretsky Effect: Higher levels of content understanding is associated with lower student outcomes, but higher adherence to program procedures boosts student outcomes
In this study, we were interested in evaluating whether Math Corps needed to consider the content knowledge of applicants when selecting interventionists for the program. We were also interested in whether interventionists who followed the program procedures closely in schools saw better results than tutors who followed the scripts less closely.
In general, we saw no evidence that tutor content knowledge fostered greater student understanding—in fact, we saw the opposite. Tutors who had higher than average content knowledge tended to see less growth among their students than tutors who had lower than average content knowledge. Conversely, we tended to see higher growth rates among tutors who adhered to lesson scripts more closely. The worst case, then, were tutors with high content knowledge but low fidelity to the scripts.
The practical implications of this work for Math Corps were to stop using (and paying for) a test of tutor content knowledge and to increase the number of observations of tutors in practice to help ensure the program was implemented as designed.