WestEd i3 Evaluations Meet Evidence Standards
This post was written by Jonathan Nakamoto, Research Associate in WestEd’s Health & Justice Program, and Tony Fong, Senior Research Associate in WestEd’s Regional Educational Laboratory West (REL West).
WestEd works with districts, states, the federal government, and other organizations nationwide to conduct educational evaluations that meet rigorous standards. For example, WestEd designed, implemented, and reported on several evaluations that were conducted as part of the Investing in Innovation (i3) Fund, which was a federal grant program designed to promote innovative educational practices.
A recent report found that the three WestEd grant evaluations reviewed by the authors that were conducted as part of the i3 Fund were implemented according to What Works Clearinghouse (WWC) evidence standards. The WWC, which is a U.S. Department of Education initiative, aims to be a “trusted source of scientific evidence for what works in education” and assesses the quality of studies that examine the impact of educational interventions. WestEd’s three grant evaluations are summarized below and highlight our continued success conducting educational evaluations that meet the field’s most rigorous standards.
The Arts for Learning (A4L) Lessons Project is an intervention designed to improve elementary students’ reading and writing achievement by integrating arts into the language arts curriculum. The study’s lead authors, Jonathan Nakamoto and Sandy Sobolew-Shubin, used a cluster-randomized trial to examine the impact of A4L on 3rd, 4th, and 5th graders’ achievement on Oregon’s reading/literature state assessment. As part of the study, WestEd randomly assigned 32 elementary schools to either receive the intervention or the business-as-usual condition. The results of the cluster-randomized trial did not show an impact of A4L on student achievement, but exploratory analyses suggested the intervention can impact students’ literacy and life skills. This study received a rating of Meets WWC Standards without Reservations, the highest WWC rating.
The goal of the Expository Reading and Writing Course (ERWC) is to improve the academic literacy of high school seniors and reduce the need for students to enroll in remedial English in college. This study’s lead authors, Tony Fong and Neal Finkelstein, used a quasi-experimental design to compare academic achievement for 12th graders enrolled in the ERWC with academic achievement for a matched comparison group of similar 12th graders not enrolled in the course. WestEd found that enrollment in the ERWC had a statistically significant positive impact on the English Placement Test, which was the placement test used by the California State University system to place students into either a remedial English course or a credit-bearing English course. This study received a rating of Meets WWC Standards with Reservations.
The goal of the Pathways to STEM Initiative (PSI) is to improve middle school science achievement using project-based science, technology, engineering, and math (STEM) coursework; extracurricular opportunities for students to explore STEM concepts; and teacher professional development. To assess the impact of PSI, the study’s lead authors, Jonathan Nakamoto and Juan Carlos Bojorquez, used a quasi-experimental design that relied on a group of matched comparison schools that did not participate in the initiative. WestEd did not find that PSI had an impact on 6th, 7th, and 8th graders’ science achievement. However, the implementation evaluation component of the study indicated that PSI may not have been implemented with sufficient fidelity to lead to achievement gains. This study received a rating of Meets WWC Standards with Reservations.
The ratings of the i3 evaluations, which were completed by Abt Associates, are unofficial at this time for the A4L Lessons Project and the PSI evaluation, because they were not conducted by the WWC. The ERWC evaluation has been reviewed by the WWC and can be found here. Given one of the goals of the i3 Fund was to increase the number of studies that meet the WWC standards, it is likely that WestEd’s two studies with unofficial ratings will receive official WWC ratings in the future.
Additional i3 Studies
In addition to the three studies summarized above, WestEd is currently the evaluator for two other i3 grants. Staci Wendt and Rebeca Cerna are evaluating the impact of Santa Ana Unified School District’s Positive School Climate Model and Linlin Li is leading the evaluation of Learning by Making: STEM Success for Mendocino County.
WestEd has also led several other i3 grants, including Kirsten Daehler’s investigation of the impact of the Making Sense of Science and Literacy professional development, Pamela Spycher’s project on English Learner Professional Learning, and the Strategic Literacy Initiative’s project for the Reading Apprenticeship program.
To learn more about how WestEd can help your organization design and implement a rigorous research design to answer questions about the impact of educational programs, visit our website or email us at research@WestEd.org.
Posted on October 1, 2018