Evaluation of the First In Math^© Online Mathematics Program in New York City

Methodologies: Experimental

WestEd researchers are examining the implementation and impact of the First In Math (FIM) online mathematics program on fourth- and fifth-grade student achievement in the New York City Public School District (NYPSD).

The FIM program is a supplemental instructional tool designed to bolster the mathematics achievement and student engagement with mathematics using interactive online games.

This research study is being funded by the U.S. Department of Education's Institute of Education Sciences (IES).

Research Questions

The following research questions are guiding this study:

• What are the individual and classroom level conditions that influence FIM program use?
• What is the impact of FIM on mathematics performance?
• Is the impact of FIM on mathematics performance more positive in classrooms served by teachers who have a high propensity to integrate technology than it is in classrooms served by teachers who have a low propensity to integrate technology?

Methodology

WestEd is conducting a cluster randomized trial, in which classrooms are randomly assigned to conditions, to examine the impact of the FIM online mathematics program on fourth and fifth grade student achievement, and to examine variation in impacts across classrooms with high and low support of technology integration.

WestEd researchers are drawing the sample of classrooms to study FIM program implementation and its relationship to student achievement from the pool of 661 NYPSD schools that include fourth and fifth grades and are not currently using the FIM program.

Teachers randomly selected for the experimental condition are receiving professional development in the technical aspects of the FIM program and how to integrate the program effectively into their instructional practice.

Teachers in the control condition are implementing their usual teaching practices. Thus, the control group represents a treatment-as-usual condition, representing what students would normally receive at schools participating in the study.

WestEd's research design also includes other methods — including case study, survey, and multilevel data analysis of state standardized assessment scores in mathematics — to study FIM program implementation and its impact on mathematics achievement.

Because program efficacy is contingent upon program implementation, WestEd researchers believe a rigorous examination of implementation is warranted.

The study of program implementation focuses on examining the individual, classroom, and school contexts that support or inhibit technology integration.

Providing experimental estimates of program impacts in classrooms with high and low levels of support for technology integration is a significant strength of the design, as there is typically substantial variation in implementation of supplemental computer learning programs such as FIM that undermines appropriate estimation of program impacts.

The outcome analyses will thus provide unbiased program impact estimates under optimal and sub-optimal implementation conditions.

Researchers are also conducting a qualitative study of a smaller sample of teachers using classroom observations and interviews to validate and explain quantitative findings and to identify factors that influence the success of the intervention.

Key Findings

Key findings of WestEd's research study on the FIM online mathematics program will be made available following the study's completion date.

Contact Information

John Flaherty Jr
562.799.5114
jflaher@wested.org