Assessing Student Readiness for Accelerated Math in Middle School
Danny Torres in Conversation With Dr. Guillermo López, Janine Penney, and Scott Firkins
Scott Firkins:
Any district, any school can give any test and try to use it to place students in the right place in the math pathway, but you cannot guarantee that those decisions are going to be correct unless you do the rigorous work of developing the assessment and then validating it. And Palo Alto wanted those guarantees.
Danny Torres:
Welcome to Leading Voices, a podcast brought to you by WestEd, a national nonpartisan research, development, and service agency. This podcast highlights WestEd’s leading voices, shaping innovations, and applying rigorous research in ways that help reduce opportunity gaps and build communities where all can thrive. I’m Danny Torres. I’ll be your host.
In today’s episode, we explore how the Palo Alto Unified School District and WestEd work together to update a process for determining middle school student eligibility for accelerated mathematics, creating broader opportunities for students interested in advancing their high school, college, and career readiness.
Joining us today are three leaders deeply involved in this work. From the Palo Alto Unified School District, we have Dr. Guillermo Lopez, Associate Superintendent of Educational Services, Janine Penny, Manager of Assessment, and from WestEd, Scott Firkins, Director of Assessment Content. Scott and his team have been providing technical assistance, psychometric expertise, and assessment development services to the district. Guillermo, Janine and Scott, welcome to the program.
Dr. Guillermo López:
Thank you, Danny. It is definitely an honor for us to be able to join and be able to share the work that we’ll be doing partnering with WestEd.
Janine Penney:
Yeah, we’re happy to share this exciting work.
Scott Firkins:
Yeah, thanks Danny. WestEd has been happy to partner with Palo Alto on this, so getting to do this is very exciting for all of us.
Danny Torres:
All right. So for listeners who may not know, the Palo Alto Unified School District is in the San Francisco Bay Area Peninsula near Stanford University in Palo Alto, California. Guillermo and Janine, what are some key characteristics of the district? And then tell us about the district’s efforts to reimagine its middle school accelerated mathematics program and how WestEd became a partner in the validation process?
Dr. Guillermo López:
Palo Unified School District is known as a destination school district. We serve a very diverse population. We obviously serve students here within the city, within the peninsula, but we also have a lot of students and families that transition to our city and our district from around the world. Having Stanford University right across the street, we serve a lot of students that are parents, are professors. Being in Silicon Valley, we serve a lot of the community that work in tech companies. And with that being said, that allows us to really serve students that come in already to our district very advanced in math, which has allowed us to make some adjustments within the reimagining math process that we have for students that want to continue to advance even further.
Janine Penney:
Many of our students love math and can demonstrate their learning at really high levels. So as a result, Palo Alto has developed an accelerated middle school math program for all students that we call Reimagining Math, which is four years of middle school math in three years. The validation assessment offers an opportunity for students that are performing at a really high level to skip one of those middle school math courses. And so for a small number of those students, they can go through our middle school math program at an even faster pace. So we’re talking about completing five years of math in three years. And so for students to be able to do that, we call that the math validation process. It’s optional. It’s open to all students entering sixth, seventh, and eighth grade, and it allows students to demonstrate that they’re ready to skip one of those middle school math courses.
Each course that they want to skip requires its own validation assessment. And this process ensures that students truly have the skills and knowledge to skip that class. If a student successfully validates a course, then they’re able to skip it within the middle school math sequence, and it can open the door to more advanced opportunities in high school, like reaching Calculus BC by their junior year or other rigorous math courses. So we were looking for a partner to help us redesign those assessments, so a partner that could help us build something or rebuild something that was valid and reliable and aligned to that condensed math coursework.
Scott Firkins:
WestEd became involved with Palo Alto, and it all started with an email. I know that Guillermo had a contact at WestEd, and he sent an email to her just sort of inquiring about getting some assistance with building a validation test that goes along with this accelerated math program. And Guillermo’s contact, her focus area was not assessment. And so Guillermo’s email ended up in my inbox, and when I read it, I immediately was like, “We can do this. WestEd can do this.” And one of the things that my team and I felt excited about even before we talked to Guillermo was that felt like there was a high probability that we could have a positive impact on students. We could develop this assessment, they could take the assessment, and then they could get placed into the correct math course within the math pathway. We also really like the idea of working with the local school district to do this.
You know, at WestEd we do a lot of assessment work. We wanted to apply our best thinking about design, development, validation of assessments to a local situation. And we like to do that with a very deep partnership. We like to develop a partnership with our clients and then work closely with them. We wanted to make sure that we could build trust with Palo Alto, that we could give them a plan, that they could see what is the beginning of the plan, how are we going to get through all the steps of it, and what is going to be the outcome of that. And so essentially that’s what we did. We got together and started talking about what are the needs of Palo Alto? How can we get you from beginning to the end? And how can WestEd continue to be there throughout the process to be a steadying force for you and to be a resource that you can go to and ask questions and get exactly what you want. In this case, an assessment that you have proof that it is doing what it is supposed to do.
Danny Torres:
Redesigning a longstanding process for students isn’t easy. What are some of the biggest challenges you faced along the way? What are some of your pain points, whether that came from logistical challenges, resistance from staff and students, or meeting community expectations?
Dr. Guillermo López:
That’s a great question. So the process for this assessment before, it was a two-day process, and that process was very time-consuming. One of the logistical challenges was number one, that it was a two-part test, and students would be undertaking that for two days or multiple days, depending on the situation. But also as far as our teachers, the test was previously developed by our instructional leads, our middle school instructional leads, which obviously are overseeing math. And their charge was to develop the test, administer the test, but also grade the test. So during that process, that impacted them and our students in the classroom because they would have to be removed from the class.
So that impacted their instructional time for teachers, but also for our students. If they’re successfully able to pass it, they’re able to skip the actual course. If a student was challenging the sixth grade math class, it would be advancing into the seventh grade math, or they were challenging the seventh grade, that would be advancing into algebra or geometry, depending on where the student landed. So that we just felt that it has to be a better way to implement this, but also to ensure that we were properly assessing our students to make sure that they were placed in the proper grade based on the skills that they had shown within the test.
Janine Penney:
Yeah, I think it wasn’t an easy process previously, and redesigning the validation test wasn’t easy. It was finding a balance between maintaining rigor and reducing the burden on staff and students. We wanted to ease the burden on the system as a whole. It was previously a huge lift for everybody, so we aimed for one day format that included some machine scored items so teachers wouldn’t have to be scoring as much. But we also wanted the rigor where students have to show their understanding, not just in a multiple choice format. We also wanted to make sure the test was fair and reliable, free of bias. So huge steps of course, so that we know that the assessment would be measuring what it’s supposed to in a fair and reliable way, ultimately so we can give kids a fair shot. If they’re taking an assessment to see if they can skip a year of math, we want to make sure that that’s fair and based on their readiness.
A key hurdle was around building trust in the fairness of the process, because accelerating five years of math in three years is a huge step for students and also for families. They’re advocating for their children. So we need to make decisions that were grounded in solid evidence, and WestEd was a key partner in that. If we’re going to spend this much effort on creating an assessment, we need to memorialize the process. And the research brief does that, detailing from the blueprint to assessment design to the trial of some of the items, cut score development, ensuring that whatever percentage or point values we set for quote-unquote “passing the test”, that those are all valid and not just arbitrarily created. And so when we share that information publicly, that should instill some trust in the community that we’re really making efforts to build a fair and valid assessment.
Scott Firkins:
One thing I want to add about what Palo Alto is doing with this transparency, all state level assessments, all national level assessments, all international level assessments have these research briefs or often called technical manuals that accompany them. And basically it is a transparent way of saying, “This is how this assessment was developed, this is how it’s administered, this is the data that backs up everything.” It is very uncommon that a district level assessment had its own research brief. So this is a really big commitment on the part of Palo Alto to say, “We’re not just going to develop this assessment for this specific purpose, but we are going to make it public, this research brief that documents every step that we took, as well as we’re going to add to that research brief as the research base around this assessment increases, and we find out more about how it’s working and we’re going to provide the community, the school, everybody documentation about that.” It’s really a wonderful thing to see a district take up such a step in terms of assessment.
Danny Torres:
The Palo Alto Unified School District Validation Test Research brief is available online at the district website. We’ll drop a link to it in the show notes. So how did WestEd help you move things forward? How did the support, especially around things like design and development, psychometrics and validation, help you build assessments that met your district’s high expectations?
Dr. Guillermo López:
From the first time when we met with Scott and the team, it was enlightening because I have never known what a psychometrician did. And when Sarah and the other members of the team talked about their expertise, and in addition to having other members that are content experts, we knew that we were going to be in good hands. And then we started reviewing what the process would look like. We also included our instructional leads or teachers in this meetings, and we just really enjoy the collaboration, and we look forward to continuing this partnership to, again, better meet the needs of our students.
Janine Penney:
Yeah, when Guillermo and I met with some other vendors before meeting with WestEd, they’d say that, “This is what we can give you. This is what we’ve got.” And we would ask about opportunities to co-develop or make some adjustments, and it’s basically out of the box. We had four years of math to assess. Four years of math in three years. So we really need to think about what we were assessing a little bit differently. And with WestEd, it was all about the partnership from the very beginning. It was just fantastic. It was a fun conversation, but also extremely professional and grounded in all the things we’ve already talked about, you know assessment design, blueprint design.
We’re going to have psychometricians evaluating the validity of the items and the results. So all of that sounded very exciting. They brought deep expertise to assessment design, so that really helped us make sure that whatever we were developing would help us identify whether a student was ready to skip a specific course and not just if they were good at math in general, something deeper than that. And they didn’t just drop a template. They worked with us closely. And as Guillermo said, it was with our math instructional leaders who are our content experts in the district. So it was a partnership between our experts in our district, the math teachers, and the experts at WestEd who are also content experts in different areas. So that combination together was a game changer. It was fantastic. Every item was reviewed, every result analyzed. It was rigorous, it was collaborative, all of those things.
Scott Firkins:
Guillermo and Janine are very kind to say those very nice things about staff at WestEd. I appreciate that. I feel the exact same way about them. They’re just wonderful to work with, very knowledgeable. I will say this, I never once thought about an out-of-the-box or template solution. Coming out of that first meeting, I was like, “This is custom. They need a custom solution. There is no specific out-of-the-box solution that I can think could work in this case.” And so it was immediate for us, We’re going to do something custom for them. They need something custom to meet their specific context.” And that’s one of the reasons why the way that we’ve worked together, WestEd and Palo Alto work together is so important. We do bring, WestEd does bring that experience and assessment development, but Palo Alto, we need Guillermo and Janine and the instructional leads to give us that Palo Alto context. That has to be there or this particular product or solution is not going to work for them.
WestEd has decades of experience developing large scale, high stakes assessments, and in this case, typically large scale, you think of statewide, but in this case it’s district wide. So it is an assessment that is not just for a classroom, but it is for an entire district. And this is a high stakes test. Individual students take this test, and their outcome on that assessment impacts where they’re placed in this math pathway. That is very high stakes for them. And when you’re developing large scale, high stakes assessments, there are industry best practices that must be followed to make sure that the test is going to do what it’s going to do. And all we did at WestEd was apply those processes. This entire process had to be based on industry standard ways in which this is done. One of the main ways that, what we leaned into was heavily relying on psychometrics, and you heard both Guillermo and Janine mentioned Sarah.
Dr. Sarah Quesen, who’s psychometrician, she’s our head psychometrician at WestEd. She just did a fantastic job in partnership with me in Palo Alto in recommending and requiring that certain things get done at certain stages to make sure that this assessment was going to work. One, she wanted to make sure that we looked at individual item performance in the trial phase. We tried out every item before we used it operationally, and her team looked at performance of every item to make sure that it could be used operationally. Once items were used operationally, she and her team looked at the performance statistics on those items again to make sure that they were doing what they were supposed to do. The psychometricians, they undertook a study to compare individual student performance on the assessment in comparison to students who actually sit for the course. How do kids who are trying to test out of the course compare with kids who actually take the course?
And then they’re also conducting a study of kids who were accelerated, how they’re doing in their current class. In other words, once you get accelerated, how are you actually performing? Are you doing what we hoped that you would do? You’re being successful, and we’re doing a study to validate that. So there’s a lot of rigorous psychometrics involved in this. And that’s one thing I don’t think that you see in a lot of district tests. Any district, any school can give any test and try to use it to place students in the right place in the math pathway, but you cannot guarantee that those decisions are going to be correct unless you do the rigorous work of developing the assessment and then validating it. And Palo Alto wanted those guarantees, and a lot of it has to do with they wanted to be transparent with their school community, but also with their community at large. And so Palo Alto immediately just committed to this very rigorous design, development, and validation process.
Janine Penney:
And I’ll add on to that, Scott. You were talking about the process of item development and then testing the performance of each of those items, and Palo Alto was involved in those discussions. All along the way, WestEd reiterated, “We’re not telling you what to do. We’re working through this together, co-developing and making meaning together.” So in those meetings, we looked at the performance levels. Sarah walked us through the whole process. We asked questions, Guillermo and myself, the instructional leads to understand how items performed, how students performed overall before we set those cut scores. So we knew that we were comfortable understanding the process and with decisions that were being made.
Danny Torres:
So I’m curious, what’s changed since implementing the new validation process, and what are you learning so far? It’s still early, but what kind of feedback are you hearing from students, families, teachers, and the community? Are there signs that the assessment and placement process are helping students take on more rigorous math coursework? Are students succeeding in those courses?
Dr. Guillermo López:
So the process, obviously is being somewhat expedited, changing it to one day, that also minimizes the impact on our teachers and students from being pulled out of the classroom for the process. So last year’s assessment, for example, it was very smooth. We really didn’t get much questions or concerns as far as from the community or students as to the process. And I think a lot of that was from having the research brief available for the community so they can see how everything was developed and the final results, how they were graded. So so far, we just completed our second year of implementing this assessment, and we’re waiting for those results, obviously, to come back. In addition to that, we’re really wanting to see how our current students that skipped last year are doing within those specific grades. So that’s the additional data that we’re also waiting to receive in order for us to add that into our overall process and to continue to make adjustments.
But overall smoother process, minimum impact on student testing as well as obviously our staff. We’re also being more intentional and mindful as to the time of the year that we’re conducting the assessments, because in addition to our validation process, we have to be mindful of the other state assessments that we are required to also have our students take. So that’s part of the conversations that we’re having as we’re also beginning to plan for next year. But as a whole, we see it as a success, and we’re really, again, just excited to continue the partnership with WestEd.
Janine Penney:
One of the biggest shifts I think Guillermo pointed on too, is that students have an opportunity to take a one test, a single focused paper and pencil validation test if they want to challenge a course. So that alone makes the process seem successful and more accessible to students. This validation test is available to all students. So it’s something that any student who feels that they might be successful, they can opt into the test. And we ensure that if students need any accessibility, if they have an IEP or 504, that we make those accessibility supports available to students. Last year, we assessed students in May, and it’s a tight timeline to get results and make adjustments and also to notify families.
So this year, we assessed in March, two months earlier, so that we would have a quicker turnaround of the assessment results so that schedules could be made and students can be placed and families can be notified. Oh, also this year logistically, the assessment took place at one school site, which contributed to a more standardized process. So that’s something that we are continually working on is the standardization. We’re also working with WestEd to ensure that those cut scores from last year, the May cut scores, do they still hold since we were assessing in March? So yeah, it’s still early days about implementing this new system, but even in those early days, there’s always some adjustments somehow that need to be made.
Danny Torres:
So for those listeners who may not know, can you explain what cut scores are?
Janine Penney:
So students take tests all the time in their classrooms. Some of them are little quizzes that might take place. Some of them might be a little bit more challenging. So a teacher might place a cut score for students. So, hey, today we’re going to give a quiz. If you get three out of four correct, you passed. So three would be that cut score. So sometimes they’re arbitrarily set, but when we’re creating a more complex assessment or state assessments, there are always cut scores, what is considered passing, or it could be other levels as well. In our case, when we’re talking about the validation assessment, we had one cut score, and it was qualified or did not qualify. Through the process of reviewing results, we determined through the psychometrician process what that level would be for qualifying for validation.
Scott Firkins:
Working with Palo Alto, the one thing I love is that they would consider what is the impact for moving the assessment from May to March? And is there reason to restudy that psychometrically about what’s going to happen with the cut score when they take the test six, maybe eight weeks earlier than they did? That’s where WestEd comes in. We’re not going to just abandon them, say, “Oh, well, we got you through this first year and now you’re on your own.” We’re not. We’re continuing to work with them and make sure that they understand impacts like that, and we can provide hard data about whether there is some sort of impact or whether there’s not an impact. But overall, I’d say that there’s a lot of data that are still being analyzed. We know the items themselves are working. We’ve done the data analysis on the individual items.
We know that the items are good, the items that we’re using are serving the right purpose, but we’re more looking at a lot of overall trends right now. We’ll say that early analysis is showing that this new assessment, this new validation process, is identifying students who are ready or qualified for acceleration and that those students are going to be successful. One of the things about the research brief is that while it was initially published, basically at the end of year one, as we get through additional years or any specific changes, we will help you update the research brief so that it remains current with what’s going on.
We can sort of summarize the performance of students who were accelerated through the validation test process, how they’re doing in their actual coursework after that, we’re going to add chapters to the research brief to give documentation of that. So today’s version of the research brief will be a little bit different than tomorrow’s version of the research brief, but that’s what research briefs are for.
Danny Torres:
So as we’re coming to the end of our time together, do you have any last thoughts for our listeners?
Dr. Guillermo López:
So I think from the get go, even up until now, it’s been, our partnership has been amazing. I feel like I’m constantly learning more as to the specifics about assessments and just also providing a very transparent process with our community has been great. And from our end, I think we continue to ensure that this process is transparent, that our students are properly being placed. We’re also excited to see the growth in our students as we have identified based on the data from fifth grade, sixth grade, and seventh grade, how some of the work that we’re doing in other grades is properly preparing our students for challenging math because it is accelerated, but it also allows us to better make those supports for students that may not necessarily be ready.
Janine Penney:
So for some final thoughts. Change works best when it’s built through strong partnerships. It’s not always easy, and we really appreciated our collective focus on fairness and clarity and what’s best for students.
Scott Firkins:
And I would say that accurate student placement is no easy feat to accomplish, but it should not be arbitrary decision. There are significant impacts on students, however they’re placed. If they’re placed properly, then they get a very good math education because they’re taking the right course at the right time. But the impact of students that are misplaced, it’s terrible. If they should be identified and they’re not, they lack access to the courses that they should, or if students are placed in these advanced courses, but they’re not ready for it. So there’s a lot of student impact, and we want to make sure that that’s not happening. And Palo Alto saw a need to make sure that students’ outcomes are addressed through this process, and they did it in a very rigorous way in all the decisions that they’ve made. One of the last things that I’ll say is that assessment is not necessarily a popular topic.
I’ve worked in assessment development for over 20 years now, and I know for a fact assessment is not always a popular topic. However, when assessment is done right, meaning it’s designed for a specific purpose and it’s utilized for that specific purpose, it is an incredibly valuable and accurate tool. And Palo Alto was willing to invest time, money, resources in developing this very specific assessment for this very specific purposes, but because of the benefit of the students, and that’s what we’re about in education, how is it going to benefit the students? And so the investment was made that Palo Alto is not just like, “Hi, WestEd, as long as you say it’s doing what it’s doing, we’ll be fine with that.” No, they’ve asked us to prove that, and we built in a way to prove that, to give them data along the way that shows that it is working in the way in which they want. So it’s been such a pleasure working with everybody at Palo Alto.
Danny Torres:
Thank you, Guillermo, Janine, and Scott for being on the program. And thank you to all our listeners for joining us today. You can find this and past episodes of the Leading Voices podcast online at wested.org/leadingvoicespodcast or on Apple Podcast, YouTube music, Pandora, iHeartRadio, and Spotify. This podcast is brought to you by WestEd, a national nonpartisan research, development, and service agency. At WestEd, we believe that learning changes lives. Every day, we partner with schools and communities across the country to improve outcomes for youth and adults of all ages. Today’s episode focused on one really important facet of the work that we do at WestEd, and I encourage you to visit us at WestEd.org to learn more. Special thanks to Ellie Kaverman for her collaboration on this episode. And to Sanjay Pardanani, our audio producer, thank you all very much for joining us. Until next time.