How Education Leaders Can Identify Trustworthy Research for Stronger Decision-Making
Marley Arechiga in Conversation with Kirk Walters, Vice President of WestEd’s Center for Teaching and Learning
Kirk Walters:
When educators are looking at all the resources out there, it’s actually staggering how many curriculum providers, professional development products are on the market, but very few of those are really backed by strong evidence.
Marley Arechiga:
Welcome to Leading Voices, the podcast where WestEd experts break down pressing challenges in education and human development–and provide insights you can apply to practice and policy. I’m Marley Arechiga.
Education leaders make high stakes decisions every day on everything from curriculum and professional development to staffing and technology investments. There’s a lot of research that can help guide those decisions, but with so many studies competing for attention, how do you know which findings to trust, especially when time is short?
Here to tell us what we need to know is Kirk Walters, Vice President of WestEd Center for Teaching and Learning. Kirk spent 10 years as a math teacher before becoming an education researcher. Over the last two decades, he’s been studying what works in schools and helping educators use evidence to make well-informed decisions. Kirk, thank you for being here.
Kirk Walters:
Thanks for having me.
Marley Arechiga:
What kind of questions are education leaders asking you to help answer today?
Kirk Walters:
One question they ask is, is this product working and how do I know? So if a state leader or district leader just purchased a product to accelerate learning for math students, let’s say, is this coaching model that I just purchased or I’m supporting, do these kinds of approaches or products really lead to improved teacher outcomes or student outcomes? What does that cost? And how can I be sure of these results? So those are some of the questions we’re getting related to evidence and whether it’s working.
Another question we get is related to implementation. So great, I’ve purchased this product, there is evidence behind it, but how big of a lift is this going to be for the average teacher in my state or in my district or in my school? A third question we get is really about community and we are engaged in some communities of practice where you have leaders who really want to learn from each other, particularly in rural contexts where some of the educators and leaders might be more isolated.
When you bring those folks together in a community of practice and involve researchers as well as product developers, those conversations tend to be extremely rich and generative, especially when educators are at the center of that collaboration because you can really find out from each other about how this research lands in the average classroom. Those kinds of communities of practice are really, really great.
Marley Arechiga:
As you work with those leaders, what have you learned about the role research can play in making better decisions?
Kirk Walters:
Yeah. Well, one thing that leaders want to know is return on investment. So it could be at the teacher level, it could be at the student level. They want to know if what they’ve invested in is delivering the results that were advertised. And they might look after one year if there’s evidence that says, “Hey, this can support kids after one year of use.” They might wait multiple years if it’s something like a professional development program where it takes longer sometimes to translate into student outcomes. That’s one thing.
Another thing that we see quite regularly is that leaders have visions that they want to enact, and many of those visions are very ambitious. And the leaders who use research to roll out evidence-based vision, and when it’s being rolled out, they’re collecting data along the way from colleagues, from educators, that just really sharpens the vision, improves coherence.
The third thing is related to credibility. So if I’m a leader that has an initiative or an idea that really is rooted in evidence, solid evidence, and I implement the initiative in ways that really do collect feedback from educators, I’m going to have a lot more credibility about this idea because I’m grounded in real data, I’m grounded in input from educators, the folks doing the work on the ground. And this is really a trust building thing. So I think that’s super important too.
Marley Arechiga:
Building on that point of credibility, how do you know when research is having an impact beyond the final report?
Kirk Walters:
One way you can tell is by uptake. So if there’s a report, it was done in a particular context in set of districts and schools, you might ask, are they still using the product after the report was published? Because if they’re still using it, it suggests that it wasn’t just the report, it was what was in the report and it’s being sustained.
Another way outside of the study report is you can see if this is an evidence-based approach, is it spreading outside the district that was in the study to throughout the state through other states? Is it accumulating additional evidence as it spreads to different contexts? So that’s an ideal kind of trajectory for a solid report. “Oh, then we tried it in this context. We made these changes and yet we still find that there are solid effects. “So that kind of spread, especially when there’s solid evidence that continues to be wrapped around that thing as it evolves into other places, that’s a really, really good sign that the uptake is evidence-based and grounded.
Another way, and perhaps this is the ultimate dream, the ultimate goal of a researcher, is that your work actually influences policy. So the decisions that legislatures make or school board makes related to evidence. Honestly, that’s why most of us get into this line of work in the first place. We want our work to actually matter and make a difference.
Marley Arechiga:
Are there any examples you can share of research projects that have caught the attention of a legislator?
Kirk Walters:
Yes. In fact, I have a current one right now. We’re wrapping up a five-year study of this really innovative professional learning network among secondary math teachers in a rural state. And these teachers have received stipends for their work that they do outside of the classroom. They collaborate, they learn together. They also serve as part-time coaches in their schools. So it’s like a, I’m a teacher, but I have a little bit of release time to do instructional coaching. It’s a really interesting model.
Our evaluation showed that there’s a retention benefit, and recently we found positive, strong associations between student learning on the state test and whether they were working with one of these mentor teachers. So it’s really great findings. This model has now been incorporated into the state, into statute essentially, so that a district could say, with some of my state dollars, I could choose this kind of professional development program and get support from the state.
Kirk Walters:
So that’s a really happy story of how you have an innovative program, solid evidence that actually makes it into a statute so that it can be accessed by other educators who might be interested in that. So that’s one recent example.
Marley Arechiga:
How can education leaders tell whether a study is trustworthy? What are some of the markers of trustworthy education research?
Kirk Walters:
So one question that an administrator, a teacher, a parent could ask is who conducted the research? So was this something that, let’s say I developed the professional development program and then I had my brother evaluate it or somebody that I knew or somebody internal to the company. It may be a great evaluation. It may lack some of the independence if you don’t have somebody from a university or somebody from an independent organization validate the results that had no stakes in the outcome.
And then when you look at that study, what’s in it really says a lot. So the methods, if you’re trying to find out if something worked and I can really trust this product before I use it in my classroom, you want to make sure that the research methods are clear. So for instance, was there a comparison group when they studied this product? Was there a similar group of students or teachers or schools that were used to compare how they were doing compared to the group that was using your product?
Marley Arechiga:
Okay. So far we have who conducted the research, what are the methods the researchers used? What else should listeners look for?
Kirk Walters:
You also want to look at the measures. So if I developed Kirk’s professional development in third grade math and I also developed an instrument as opposed to using an instrument that’s been validated, that maybe it’s the National Assessment of Education Progress, the NAEP scores, which we know are used as kind of the nation’s report card, there are publicly available NAEP items that I could have used to test my study or I could have just come up with my own items. A NAEP based test would be stronger because you know that those items have really been validated, they’ve been used widely, they’re measuring what they say to measure. Whereas if I just came up with my own measure, it’s probably not going to be great the first time I designed it unless I’m a psychometrician.
I also think where the study was done matters. So if you’re a teacher in Southern California in a certain community that has different kinds of issues, challenges, opportunities and that’s your school, that’s your district, and you’re looking at a study that was done, maybe you’re an urban area, in a remote rural place with a different community, it may be applicable, the results may apply or you may be more interested in finding a similar kind of community to sort of see if some of those results are the same.
So you can look for things like were the students performing at similar levels of math and literacy, for instance. You can look at things like income and opportunities and employment figures and things like that to make sure it’s a similar context. I would say that the clearer the methods are in the more closely aligned the sample and the setting is to where you’re doing your work, you can have more confidence that those results are going to apply to you and to your context.
Marley Arechiga:
How does education research support better decision making in schools?
Kirk Walters:
One way research can help is by supporting a teacher or a principal to become a better consumer. When educators are looking at all the resources out there, it’s actually staggering how many curriculum providers, professional development products are on the market, but very few of those are really backed by strong evidence. And it could be that you’re interested in a newer program that hasn’t had time to accumulate evidence. That’s common, but you can still look at the design features of that program and see if they’re aligned with other programs that do have stronger evidence is one workaround there.
Once you adopt an evidence-based program, it’s critical, but only the first step. How that product or program is implemented is crucial. So when you’re looking at the product, when you’re looking at the implementation model, sort of what is intended for this product to work, then think about how feasible it would be to do that, to implement that program in your school, in your district before you buy it. So I think there’s a feasibility question, there’s an intended use question that is connected to the evidence, but overall I would say good products as well as good studies don’t give implementation short shrift.
Marley Arechiga:
So where do listeners even find the kind of research you’re describing? Where do you recommend that they start?
Kirk Walters:
Maybe they’re talking to their chatbot these days, but if you were to put in, and I have done this plenty of times, but when you put in high quality research or rigorous evidence or effectiveness, things like that, a lot of the sources that come up were federally funded, not exclusively, but the Institute of Education Sciences, IES is the research arm of the US Department of Education. You can find a lot of solid evidence on the IES website that represents almost 25 years of studies. I think other people would look for universities, potentially universities where they have a prior relationship or a connection as a source of what kinds of working papers are respected universities putting out.
So I’d say universities, the federal government, not just IES, but other federal agencies, the National Science Foundation, the Office of Elementary and Secondary Education, OESE has its Education Innovation and Research program and it’s a huge investment in testing whether innovations work.
And so there are reports that are associated with pretty much any topic related to education. So there’s a lot of good stuff out there if you know where to look, or I suppose if you have some of the right keywords.
Marley Arechiga:
A lot of studies out there are long or really technical. What are your suggestions for education leaders who are trying to vet research but are short on time?
Kirk Walters:
Yeah, I mean, they are going to be short on time. I think that’s one thing we’re sure of. I would be surprised if they weren’t short on time, but I think it’s a really good question for that reason. I do think that a really good abstract has enough about the design in it that you can kind of tell if it’s worth reading the whole thing or digging in more. So I do think there’s something to be said for screening abstracts from well-respected publications. They don’t all have to be peer reviewed journals. Some of those are paywalled, but some of the other examples I gave, like a working paper from a respected university, The Hechinger Report summarizes research in an email, a newsletter where you can… And they’re a nonpartisan education journalism organization, nonprofit. And I find that their summaries of research weekly emails are on important topics and they’re also distilled in a couple different grain sizes.
And the more you read publications like that and you realize that what goes into it, to your earlier question about trustworthy, you can count on those studies having met a certain bar. And then circling back to whether it’s IES or OESC, some of the federally funded projects where they have examples of reports, there are often infographics that are essentially executive summaries or you can maybe have time to read the executive summary before you read the whole report. And then the other thing is I’ve found that authors really like it when you email them about their work. And so if you have questions, often the author’s contact information is there and they would actually love to be contacted usually and could give any additional detail. If you had questions that needed to be clarified, that’s a good way to do it too.
Marley Arechiga:
Yeah, go straight to the source.
Kirk Walters:
Yeah, exactly.
Marley Arechiga:
I wonder also about a quick scan for buzzwords. Are there any specific phrases listeners should look out for when they’re evaluating research studies?
Kirk Walters:
So if you hear someone say, “This is effective,” “It’s effective because,” that actually means something pretty specific in research. So it implies that there’s causal evidence that this thing caused this outcome. And in order to get there, you have to have used a randomized control trial or a quasi-experimental study where you have a treatment and comparison group and they’re matched and they’re very similar kinds of groups before you can say anything about effectiveness. And so when people say, “No, this totally works,” be a little suspicious, look at the design a bit more. Also, you can often, and this is fine to say this, if students grow over a semester from pre to post, so let’s say they’re in fourth grade and they take a fractions assessment at the beginning of fourth grade, and then at December you might see that there’s growth and maybe you taught fractions differently than you had in the past.
Just reporting pre-post is not good enough because we all learn every year, kids learn every year. So you don’t know if that growth was attributed to the way you taught fractions or if it was just natural growth, which is why having a comparison group accounts for that because then you can see, oh, the comparison group grew this much and they didn’t have my amazing fractions intervention, but the treatment group did. So I would say pre-post doesn’t mean that it was an improvement that you can attribute to that thing that you did. And then I would also say just watch out for the word effectiveness.
Marley Arechiga:
I wonder about operationalizing this kind of work. Are there systems or roles that school leaders can create to help them vet research?
Kirk Walters:
Yes, I think there are. And I mean, there are plenty examples of where school systems, whether it’s a school or a district or even a classroom or a department where they just take evidence seriously, whether it’s continuous improvement efforts that they’re trying to implement in their classroom or as a department, or if it’s a school-based program where everybody is connected to why we’re doing this thing, hopefully it’s an evidence-based thing, but you’re pulling everybody together in a coherent way so that you understand the model. I have seen leaders who say we’re all going to learn about this approach because coherence matters, let’s say. If we have a tier two literacy program that students experience three times a week with a paraprofessional and this tier one program that students are there every day, if there’s no coordination between the tier two instructors, the paraprofessionals, the adults in the building, then that coherent approach is not going to roll out in the way that it’s intended.
And a lot of the research behind really any intervention, there’s guidance for how to use it. And I think when you see school systems, leaders, teachers that are serious about trying to use it in the way that it was intended because it’s grounded in the evidence, it looks different and it feels different.
Marley Arechiga:
So maybe it’s not so much top down, it’s more about getting the right people in the room and coming to some sort of collaboration, discussion, consensus together before we really get into implementing that research. Is that what I’m hearing?
Kirk Walters:
Yes, I think there’s that and they’re setting the stage in the right way. And maybe it’s something the principal didn’t pick, but maybe it was a representative committee across the school that collaboratively decided on this thing. And then once we decided to use it, there’s ongoing opportunities to see how it goes when it’s implemented because even if you have the best, this program with the strongest evidence base, it’s probably going to land six different ways across their six different classrooms. So I think the vibe is one of let’s use high quality stuff and then let’s create high quality learning opportunities to continue to learn on how it’s working.
Marley Arechiga:
Okay, Kirk, we are coming to the end of our time. Before we go, can you give us some suggestions for building a culture where decision-making is based on evidence?
Kirk Walters:
I think curiosity is the key, and if you can create enough space for genuine dialogue around important questions, having the right resources, and just enough of a bandwidth where people in the building, let’s say I’m the principal, people know that the principal is really interested in using the best evidence we can. We’re all doing it together. When you see that, it’s contagious. Just super quickly, I remember recently a principal who had adopted an evidence-based high quality instructional math program, but when he did his walkthroughs, he would sit down intentionally and do the problems with the kids because he said, “When I am doing those problems, they’re supposed to be problem-based. They’re supposed to be grappling with the concepts. I want to experience it.” And then he takes that to his staff meetings, he takes that to his walkthroughs, and those are sort of powerful models.
So a principal or an administrator who says, “Hey, I just made it through the study. I read the executive summary. I have three questions. What do you all think?” It can be done if it’s prioritized.
Marley Arechiga:
Thank you, Kirk. We will wrap it up there. Thanks for being on the show.
Kirk Walters:
Thanks for having me. This is super fun.
Marley Arechiga:
Yeah, of course. For our listeners, check out the companion checklist for this episode and the show notes. It’s a handy compilation of the advice Kirk shared with us today. Use it the next time you’re digging into research that will inform your next big decision for students. You’ll also find more information in the show notes about how to partner with WestEd on research or evaluation projects, including our research practice partnerships. Feel free to keep the conversation going with Kirk directly on LinkedIn or by emailing him at [email protected].
While you wait for the next episode, subscribe to Leading Voices on Apple Podcasts, Spotify, YouTube Music, Pandora, or iHeart.
This podcast was brought to you by WestEd. WestEd is a nonpartisan research development and service agency. We work to promote excellence, improve learning, and increase opportunity for children, youth, and adults. Special thanks to our audio producer, Sanjay Pardinani, and to Gretchen Wright for her editorial support.
Thanks so much for tuning in. We will catch you in the next episode.