REMIQS Project and State School Accountability System
Danny Torres in Conversation with Dr. Raifu Durodoye.
Dr. Raifu Durodoye:
We’ve essentially created a blueprint for these state education agencies to get a sense of how they could improve their data systems, how they could think through changing or modifying their current accountability frameworks, and how, by identifying particular student groups have been historically underserved, and getting a better sense of their experiences and their outcomes in schools, all those things can really team to improve the system.
Danny Torres:
Welcome to Leading Voices, a podcast brought to you by WestEd, a national nonprofit, nonpartisan research development and service agency. This podcast highlights WestEd’s Leading Voices shaping innovations and applying rigorous research in ways that help reduce opportunity gaps and build communities where all can thrive. My name is Danny Torres. I’ll be your host.
Today we’re here with Dr. Raifu Durodoye. Dr. Durodoye is a senior research associate at WestEd. His work focuses primarily on evaluations of education programs and interventions. Dr. Durodoye also provides technical assistance to school districts and state education agencies. Most recently, he’s worked with KnowledgeWorks on the REMIQS Project, which is a multistate, mixed-methods investigation designed to explore how we can best serve our most vulnerable learners.
Today’s topic: The REMIQS Project, and a look into comprehensive accountability systems that can help improve outcomes for systemically underserved students. Dr. Durodoye, it’s great to have you with us on the fourth episode of our podcast.
Dr. Raifu Durodoye:
Thank you, Danny. It’s great to be here.
Danny Torres:
Now, before we dive into today’s topic, tell our listeners what inspired you to get into the work that you’re doing now.
Dr. Raifu Durodoye:
No problem. I will start with my upbringing. My father’s an educator, my mother’s an educator, so from a very young age, I think it was almost a foregone conclusion that I was going to be in education in some form or fashion.
There was a time when I was in high school speaking with some classmates about their prospects for going to college, and me being very idealistic and excited about my future and my plans was speaking with them, and basically, one of my classmates had told me they really didn’t want to hear about that, simply because of the fact that they knew they weren’t going. And, that really crystallized in my mind the fact that there are some students that they don’t have the access, they don’t have the ability, and they’re not going to really be able to realize their dreams educationally simply due to their background or their circumstances. That really motivated me to work in education to try to address that, and to support all learners to meet their full academic potential.
Danny Torres:
Let’s talk about the REMIQS Project. Because when I think of remix, I think of a music producer taking a song and then creating something new, taking it into a new direction, adding and removing, changing parts. Can you tell us about WestEd’s partnership with KnowledgeWorks? How did it come about, and what are the goals of the project?
Dr. Raifu Durodoye:
That’s an adept analogy. But, the way that the REMIQS Project started, it actually started with KnowledgeWorks partnering with the Urban Institute on some thought-partnering work and some initial analyses. Then, the work was actually transferred over to WestEd, where we took over the quantitative filtering phase. Essentially, what we did was we collected a lot of data from a number of states from across the U.S., we developed our own school quality model that emphasized more equitable measures of school quality, and then we identify schools that are providing value to historically marginalized students.
The aim or the thrust of the whole project was to identify schools that are providing value to historically marginalized students, to go into those schools, and to get a sense of what were the activities, what were the features of those schools that promoted or supported their effectiveness? Then, ultimately, to disseminate those practices to schools that were underperforming, so that we could basically support historically marginalized students across the country. That was the aim, and that’s where it started. Now, we’re actually in the third phase, where we’re engaging schools, having those conversations, and trying to get a really in-depth understanding of the things that make them successful.
Danny Torres:
When we’re talking about historically underserved students or historically marginalized students, what do you mean by that?
Dr. Raifu Durodoye:
Yes, so, for the REMIQS Project, there are basically four student groups that we identify that we’re characterizing as historically marginalized. Those are: students of color; low-income students; students with disabilities; and English learners. So, for the purposes of the research study and our analyses, we’re focusing on those groups.
Danny Torres:
What are you learning about improving outcomes for underserved students or historically marginalized students?
Dr. Raifu Durodoye:
Through the course of the project, we’ve learned a lot. Again, the project is really focused on district- and state-level policymaking. From that perspective, one of the things that we’ve learned is that we need to be better users of the data that we have available to us. That means districts and states could actually really support these efforts by investing in state longitudinal data systems. Those are essentially data systems that track students over time, and they can give us the ability to evaluate educational interventions and strategies in the long run. That’s one thing that we can do that we’ve learned from this project.
Secondly, there is a need for us to identify schools that produce better outcomes among our most underserved students. We’re learning that many of the schools that we see rated are basically rated based on the fact that they serve a particular demographic of students. At least, we know that these ratings are highly correlated with the demographics of the students that they serve. So, we understand that those perceptions are often very important in terms of supporting communities, in terms of the housing situation of the individuals that live in those communities, and also, just the general perception and support of those schools. Again, we’re trying to raise this awareness around the importance of school accountability, so that people have an understanding of its implications in the long term and moving forward.
And then lastly, I’d say that we’re learning how important it is to really hone in on school accountability to affect policy and practice. There’s a lot of great information that we could actually utilize if we were to make these investments, especially in terms of learning about the long-run impacts of COVID, how to better support students and schools in underserved communities, et cetera. Maybe it’s just the utility of this information to support schools more holistically is another big takeaway that we glean from the project.
Danny Torres:
Great. What role do school districts and state agencies play in this?
Dr. Raifu Durodoye:
Well, they play a very important role because—I’ll start with the state. The state is usually the arbiter of these accountability models. They’re the ones who bring stakeholders together from across the state to think about—what are the important measures that we want to make sure comprise these, composite, or these—I don’t know—composite score measures are basically these aggregated measures. They have a role in determining what’s going to be utilized, and what’s going to be communicated to the field, and also the parents and families.
School districts are also important to these conversations because, number one, they’re important stakeholders in those conversations. And secondly, we know that they’re ultimately the entities that are being evaluated, right? They’re the ones who have to make sense of and communicate to their stakeholders—parents and families—what these different accountability measures mean and what the implications of those measures are for the students that they ultimately serve. They’re very important, and they also—again, maybe lastly, and for some, most importantly—they have a tremendous impact on school funding.
Danny Torres:
All right, so, I want to shift and talk about accountability models, because I think it relates to the work that you’re doing with the REMIQS Project, as I understand it. What is an accountability model, and how can we ensure that those models are equitable for students?
Dr. Raifu Durodoye:
An accountability model is basically just a way of waiting, or prioritizing different educational measures, and then, in turn, using them to quantify the effectiveness or the quality of a school. Summarily, I guess you could say we just utilize large amounts of school data to provide insights on school quality.
I think when you get down into the technical aspects of them, we derive ratings from accountability models based on formulae that are developed by the state in partnership with stakeholders in districts and in the community. States collect the data, districts collect the data, it’s input into these formulae, and then we’re able to rank-order schools in terms of what we deem are the best schools, to all the way down to what we would deem underperforming schools.
Danny Torres:
I think how we define school quality matters. In the context of the REMIQS Project, how are you all defining quality?
Dr. Raifu Durodoye:
I will point to two ways that the REMIQS Project differentiates itself from maybe conventional notions of school quality. I think the first way is that we’re homing in specifically—if I can get that word out—on historically marginalized students. Oftentimes, when we look at the conventional accountability models, we’re looking at all students, which is not an issue. But, the problem with that can be that you’re shedding the experiences and the outcomes associated with particular student groups. We want to make sure that that no longer happens, and we have information about the experiences and the achievement of all students within the school.
I would say, secondly, what we attempt to do is to expand our understanding of school quality in terms of the measures that we use that might be indicators of whether or not a school is high quality or low quality, for lack of a better term. When we think about those conventional measures, the test scores, the graduation rates, those are all fine, and we don’t argue that those should be necessarily omitted from accountability models. But, what we would argue, is that to really remix—to really recast and rethink the way that we do this—we need to include some additional measures, things that also relay important information around the effectiveness of the school and the value they provide to students once they graduate.
Danny Torres:
What are some examples of those measures?
Dr. Raifu Durodoye:
Some examples of the additional measures that we think need to be included in accountability models are things like civic engagement—whether or not you vote once you actually graduate from high school. There are also things focused on economic mobility—whether or not you’re able to gain employment once you graduate, and whether or not you’re able to secure a living wage.
We’d also say things like social-emotional learning outcomes, which are gaining more speed. We know those things are important, just in terms of being a contributing member of society. And then also, postsecondary enrollment, which is an indicator that is starting to come into the fray in more discussions and conventional school quality measures, but not always, and something that we think should always be taken into account when you think about whether or not a school can support you in your path—your trajectory to postsecondary education.
Those are a few other things that we’re trying to introduce to the field to think more holistically about accountability models and school quality.
Danny Torres:
Where can we find more information about the accountability model from the REMIQS Project?
Dr. Raifu Durodoye:
In our work with the KnowledgeWorks Foundation, they’ve been great partners, and have developed a website that has a lot of information around the project’s background, the different phases of the project, where we are now currently in terms of completing the project, and then, some of the aims and the goals of the project in the long term, in terms of really having an impact in the way that states do accountability.
Also, we’ve done a number of blog posts that basically summarize our thought processes, our thinking in the theoretical underpinnings of the REMIQS Project. Those are available to anyone who’s interested or inclined in getting more information around our thought process or the information, the evidence, that we’ve gathered to suggest that this is an important project for this moment. And, for anybody who’s inclined to look at the more technical aspects of the work that we did, you can also find links to the technical manual that we produce that summarizes the statistical methodology, our findings, descriptives, et cetera.
Danny Torres:
Great. Is there a website that we can point people to?
Dr. Raifu Durodoye:
Yes. If you go to REMIQS.org, that will be a great springboard to get to all the content and also the blog posts that I mentioned. REMIQS is R-E-M-I-Q-S, as in, Robust and Equitable Measures to Inspire Quality Schools. The play on words, it is a remix, but not necessarily spelled exactly like remix.
Danny Torres:
Again, that website is REMIQS.org. That’s R-E-M-I-Q-S.org. Can you tell us a little bit more about how accountability models can affect schools, communities, and students?
Dr. Raifu Durodoye:
I’m thinking back to my experiences as a Title I program manager at the Delaware Department of Education. Just even thinking around community support for different schools, and going to different schools, knowing their rating, and understanding the community support, the support that we gave them as a state, and the way that we engaged those schools, all predicated upon how we rated them, these accountability models—they do have implications for funding, for autonomy, for reputation, and even for community engagement.
Given that we know many of these accountability ratings that we’re often using don’t really give you a full picture of school quality or a full understanding of the effectiveness of the school, it always made me think: We can do a better job of this. We can get better information to inform better policies to support these schools a bit better.
We’re in a situation where we utilize these rankings to shame and to penalize schools, as opposed to utilizing them to better understand schools and to support them. I think that’s one of the ways that we could use the REMIQS Project to rethink, reconceptualize the way that we use accountability ratings so that they could be a benefit to all schools, as opposed to a tool to penalize them.
When we think about the implications for students, it’s really filtered through their parents, who utilize this information to make very high-stakes decisions with regard to where they’ll be willing to send their child to learn. Again, when we think about the patterns, or the fallout, or what stems from downstream from these models, it’s one wherein schools that serve large proportions of students of color, schools that serve larger proportions of low-income students, are the schools that are losing a lot of investments. They’re using reputational bearing, they’re losing investments in the community, creating a pernicious cycle wherein those schools are actually left with less than they had before, all based on the fact that they’re being evaluated using a measure that doesn’t really give us a true, comprehensive understanding of the work that they’re doing to support students.
Danny Torres:
Right, so, it sounds like it not only impacts the students’ experience at school, but also their long-term outcomes.
Dr. Raifu Durodoye:
It does. It does. Even thinking about my time working in higher education, we knew that when students came to us from particular schools, just based on their reputation, based on the fact that we knew historically they had been low-performing, they were treated differently. Sometimes, in a way where we thought, you know what, maybe we need to provide those students with more support. Sometimes, in a way where people had lower expectations for those students, simply based on the school that they went to. The idea is that this isn’t information that is accurate or precise enough to be making many of the decisions that we do make using accountability information. The hope is that, through the REMIQS Project, we can provide some finer-tuned, more granular, more specific, more rigorous information to make policy and practice decisions around supporting schools.
Danny Torres:
What challenges around reframing—or remixing—accountability systems do you face doing this work, and how do you overcome them?
Dr. Raifu Durodoye:
That’s a great question. I think the first is that, when you talk about changing accountability systems often housed at the state level, we have to recognize that that’s a big undertaking, right? There’s been a lot of time and a lot of money that’s been invested in the systems that are in place, and what we’re proposing is a total recasting of those systems. There’s just some institutional inertia that you have to combat in order to make a change like the one that we’re proposing, but we think it’s worth it, and we think that it makes sense to support students.
I think what’s also important to be cognizant of is the political opposition to this type of work. Wherein discussing student equity was not outside the norm at one point, just given my experience in education, now those conversations have become politicized to some degree. It’s just more difficult to navigate spaces where you want to talk about supporting students, equity, and providing supports to the students that need it the most. It’s simply not framed, understood, or perceived the same way.
And then lastly, I think it’s a marketing challenge. I think accountability, even the word, it sounds like you’re in trouble. I mean, if you say you’re going to hold somebody accountable, it’s usually not a good thing. When we think about reframing, or thinking about accountability differently, and going to this notion of identifying high-performing schools so we can disseminate best practices, it’s incumbent upon us to help people understand we’re not trying to do the same. We’re not trying to penalize. We want to understand in order to support, in order to improve, in order to move the sector forward.
I think those are the three main challenges that we need to be aware of that we’re really, I wouldn’t say fighting against, but that we need to address in order to see through the original aims of the REMIQS Project.
Danny Torres:
What are some of the things that the REMIQS Project’s doing to overcome these challenges?
Dr. Raifu Durodoye:
We’ve essentially created a blueprint for these state education agencies to get a sense of how they could improve their data systems, how they could think through changing or modifying their current accountability frameworks, and how, by identifying particular student groups have been historically underserved, and getting a better sense of their experiences and their outcomes in schools, all those things can really team to improve the system.
What we’re trying to actually do through the course of the REMIQS Project is to be an exemplar of the type of system that we like to see. Even by identifying schools, and then going to those schools, having the conversations, getting a sense of the features, the fixtures, the practices that they’re implementing to be effective, and then creating briefs, toolkits, and manuals to disseminate the information to the field—again, we want to show how that’s done. Not to say that we’re doing it perfectly, but we want to be an exemplar. We want to show that this can be done, and it can be used as a tool to uplift schools that are struggling. The hope is that state education agencies can follow that lead, learn from some of the mistakes that we’ve made, learn from also a lot of the evidence that we’ve generated, so they can start improving their systems and incorporating some of these practices that we’re recommending through our learnings through this work.
Danny Torres:
All right. Well, I think we’re hitting time, so I just want to give you a chance to offer any last thoughts that you’d like to share with our audience today.
Dr. Raifu Durodoye:
I guess the first thing I’d say is thank you for having me. I really appreciate being able to share what’s happening with the REMIQS Project. I would say, maybe, just to leave for the audience, that this is a great conversation, but again, the conversation is just starting as it relates to accountability. We’re trying to push this out to the field, but for you to consume this information, to understand and to be an educated consumer, this information is what we really like to strive towards.
Any type of pressure, or any type of questions, or any type of engagement that the audience members can have to foment this conversation, to bring it to bear to policymakers, to school district officials, to state education agency leadership, that would be great in terms of reaching the ultimate goal, which is ensuring that we have in place systems that adequately support all learners, that are understanding and inclusive of the experiences of all learners, and that really motivate positive educational outcomes for all students in schools. I’ll just leave it at that. But again, thank you, and I had a great time.
Danny Torres:
Thanks for taking the time out of your busy schedule to talk with us. And, thanks for your leadership in this work. Where can our listeners find you online, and where can we find out more about your work?
Dr. Raifu Durodoye:
You can find me on LinkedIn, in terms of my social media. If you’d like to contact me, you can feel free to reach out through the WestEd.org website. You can there see my bio, some listings of some of the previous work that I’ve done, and obviously, just feel free to reach out. If you have any questions, I’m always interested in having conversations with individuals who are invested in this particular policy area who like to bring good ideas to bear. That’s always an option, and I’m always available.
Danny Torres:
Dr. Durodoye, again, thank you very much, and thanks to our listeners for joining us. All the resources mentioned in this podcast will be available online at WestEd.org or in our show notes on Apple Podcast, Google Podcast, Stitcher, and Spotify.
This podcast was brought to you by WestEd, a national nonprofit, nonpartisan research development and service agency. At WestEd, we believe that learning changes lives. Every day, we partner with schools and communities across the country to improve outcomes for youth and adults of all ages.
Today’s episode focused on one really important facet of the work that we do at WestEd, and I encourage you to visit us at WestEd.org to learn more. A special thanks to Tanicia Bell, our content manager for the Leading Voices podcast, and to Sanjay Pardanani, our audio producer. Join us next time on the Leading Voices podcast. Thank you very much. Until next time.