Improving Ed Tech Through Research: Q&A with Michelle Tiu

February 28, 2019

According to the editors of Education Week: “Parents and corporate America are clamoring for schools to move more quickly to embrace a high-tech vision for education. And the fast-changing landscape of educational technology only complicates the task for policymakers and administrators who seek to make ‘smart’ decisions about how to proceed.”

Perhaps surprisingly, that observation was made more than 20 years ago, originally part of the periodical’s first annual report about the state of education technology. With today’s ever-expanding uses of technology in schools, leaders seeking to make smart choices about ed tech continue to face enormous challenges, which may be related to tech growth coming as part of a societal trend rather than being driven by any evidence base for what works in education.

To help build such a base and provide the kind of guidance needed in this field, WestEd’s Science, Technology, Engineering, & Mathematics (STEM) program conducts research and provides supports aimed at ensuring the development of tech tools that are practical, of high quality, and well suited to educational environments.

To capture some of what is being learned from this work, R&D Alert recently spoke with Michelle Tiu, Senior Research Associate in WestEd’s STEM program. Tiu has extensive experience evaluating and supporting the development of ed tech tools through the federally funded Small Business Innovation Research (SBIR) program and through leading WestEd’s research partnership with NewSchools Venture Fund, which promotes innovation and effectiveness in K–12 ed tech.

How would you describe your work in ed tech?

Michelle Tiu: The ultimate goals are to improve learning for students and to support teachers. WestEd’s ed tech team does this by helping developers improve their products. We help them understand what works, what doesn’t work, in which contexts, and for which learners. We use a range of research studies, from formative to summative, to help developers make ed tech tools that are more usable and effective.

What are some of the challenges you and colleagues are trying to address?

MT: Too often, districts purchase tech tools they never use. Why? Sometimes teachers were not involved in the decision-making. Or maybe the product was not easy to pick up and use, so students tried it a few times, then gave up on it. Developers might not have done adequate feasibility testing to understand how the product works in authentic educational settings. We consider research to be an essential part of a product development process that can help alleviate these kinds of challenges, and we have years of experience to back up this perspective.

R&D Alert focused on some of your work in an article a couple of years ago. How has this work developed since then?

MT: The work has expanded. WestEd has worked with more than 80 companies through the NewSchools Ignite accelerator and has served as the research partner on more than 25 SBIR grants funded through the Institute of Education Sciences, the National Science Foundation, and the National Institutes of Health. We’ve worked with all different types of companies — from small (1–2 employees) to large, in different subject areas (STEM, English language arts, social-emotional learning), using different platforms and hardware and different types of products (teacher-facing and student-facing, apps, websites).

With NewSchools Ignite, we launched an Ed Tech Research Portal to highlight and disseminate the work from our partnership. The portal features resources for entrepreneurs, educators, and funders, and includes literature reviews, webinars, rubrics, and toolkits developed from the research partnership.

What kinds of challenges do you face as researchers trying to support the improvement of ed tech?

MT: Ed tech, by nature, pushes us toward fast time lines with rapid turnaround, which is not always compatible with some types of research. Seeing meaningful gains in outcome measures may not be practical in short time frames. Developers and researchers need to adapt to this culture in a way that makes sense for both the research and the products.

Determining the right measures to use in evaluating ed tech can also be challenging. There are well-established ways to measure learning in subjects like English and math, but typically not for the kinds of subjects that many ed tech products aim to help students learn — like engineering or coding, or 21st-century skills like collaboration. And to compare the impact of people using ed tech tools against that of comparison groups not using these tools, we often need to rely on traditional pencil-and-paper assessments, which are not always the best at measuring what people might be learning through high-tech, interactive, personalized platforms.

How does WestEd’s work address these kinds of challenges?

MT: We collaborate with developers early on to help them improve their processes and their end products. We use a continuum of research, from formative to summative, depending on the needs and context and stage of a product’s development. One key element is to create a logic model that defines a product’s aims, how it is intended to achieve those aims, and what data or evidence will indicate effectiveness. We’ve written about and created a webinar on logic models. We also think it is important for the field to continue developing measurement instruments that are better aligned to the intended outcomes of products.

How can research help to improve the ed tech that is in schools?

MT: One of the most important things we focus on with developers is making sure their products are evaluated in a variety of real settings. Just because a tool works well in one context does not mean it will work in another. Some developers skip right over feasibility testing, or they work just with their “go to” school and get only a limited perspective from a few teachers who are already on board with their work. Instead, we help make sure developers’ products are accessible and effective for the particular teachers and students they are trying to reach.

Similarly, if districts or schools choose based on which product is cooler, flashier, or more unique-looking, then research isn’t factoring into the equation as much as we would want. Instead of choosing a product based on knowing someone else uses it in another school, establishing the product’s fit for your particular context and needs is essential. The question to ask is not just whether it works, but in what context does it work? Under what conditions? Has it been effective with the same kinds of students, classrooms, and conditions that your school has?

What are some key takeaways from your work that apply to developers and perhaps to end users?

MT: Our work has identified several best practices: For ed tech to be effective, teachers need to be supported in using products. Products that work best tend to be streamlined, relatively easy to use, and flexible, allowing for customization. Student-facing products need to be developmentally appropriate. All ed tech products need to be able to fit in authentic education settings, which have unique demands for technology.

One way that we help ed tech companies with these best practices is by creating rubrics for evaluating and refining their products in development. The rubrics cover those best practices I just mentioned and focus specifically on things like the academic content covered with the product, standards alignment, and its use of principles of learning science. They also focus on issues of bias and sensitivity, attention to social and emotional learning, how engaging or inspiring the product is, as well as its usability in real classroom settings. The same areas of focus are what end users might look for when selecting products as well.

What keeps you motivated to do this work?

MT: I’ve always gravitated toward technology and — like it or not — ed tech is the way education is heading. Tech is the future. It’s impacting all facets of our lives and our schools. My job and the work of my colleagues is to do all we can to make sure the tech that gets into the hands of teachers and learners is purposeful, engaging, and as effective as possible at improving education.