WestEd
Hot Topic:WestEd
Hot Topic:(First published Spring 1998)
Faced increasingly with a "change it or lose it" message about public education, states are adopting a get-tough approach: results-oriented school accountability systems with teeth.
Some 32 states and 34 big urban districts now have accountability systems based, in part, on test scores. (1) Theoretically, these are centered on high standards. They entail consequences for outcomes. They aim to elevate system performance and, ultimately, improve student achievement. Whether the accountability movement will achieve those desired results remains to be seen. Already clear, however, is that data deriving from efforts underway are bringing into bold relief the issue of chronically low-performing schools and, more to the point, questions of what states can do about it.
As states move to intervene with such schools, virtually all face the dilemma of the far extreme: schools at the very bottom that dramatically, persistently fail their students. Evolving from this dilemma are experiments with academic takeover. Used historically for fiscal crises— cases of graft or malfeasance— takeover once implied a fairly straightforward process of removing corrupt officials. What’s new is takeover for reasons of students’ failure to achieve. And here what to do is far from clear.
As an example, the 72,000-student Cleveland school district was taken over in 1995 by the state of Ohio for "severe instability," both fiscally and academically. In just two years, the district’s record $152 million debt was brought under control— "a piece of cake" compared with turning around the academics, says Richard A. Boyd, who served as Cleveland’s first state-appointed superintendent. Despite multi-faceted efforts to improve student performance, only 20 percent of 9th graders passed a recent state proficiency test. Indeed, only 75 percent of students show up on any given day.
Clearly, the answer is to get at the problems earlier, before schools reach the desperation point. As states like Ohio are painfully aware, academic takeover is largely a leap into the unknown. No body of research yet exists to provide guidance. Most state education officials admit being far from eager to step in and run an academically failed school or district. The record of attempts is scant; of success even more so. The issue is bound up in questions of why schools fail, who is really responsible— for both the low performance and the turnaround— and, most fundamentally, how to effect a turnaround. Involved are issues of organizational behavior, community dysfunction, human psychology, legal precedents and larger, contextual problems of race, class and urban neglect. Yet the state ultimately bears responsibility for acting on behalf of the kids in such schools, almost invariably poor and minority children whose very life chances depend on the quality of their schools.
Hence, the upsurge of interest in early intervention— stepping in with state sanctions and/or assistance at the first signs of trouble. To do this, states must be absolutely clear about expectations of schools, careful to define the continuum of possible actions (e.g., stages of intervention), concise about what will trigger academic intervention and when, specific about the kinds of support to be provided at each stage, and— importantly— clear about what indicators will offer sufficient evidence of progress. The state can then put all its energy and resources into making sure that takeover, the option of last resort, never has to happen.
Under this approach, the threat of intervention — including, possibly, takeover— is harnessed as a motivational force, in the vein of Samuel Johnson’s quote, "When a man knows he’s to be hanged in a fortnight, it concentrates his mind wonderfully." School staffs obviously chafe at having their school publicly branded— a practice being dubbed "accountability by humiliation." But experience so far indicates that the savvy and dedicated among them may welcome the external threat as a long-needed catalyst for gathering all forces in the school and greater community together under a flag of common purpose and— equally important— a deadline. The state, meanwhile, eager to avert takeover situations, has a strong incentive to go well beyond enforcer, instead partnering with the school and district to oust complacency, marshal combined talents and foster innovation and risk taking.
This paper looks at some of the difficult issues states are working through as they set up and implement strategies for academic intervention. What "machinery" needs to be in place to enable and support a program of state intervention with low-performing schools? What are the nuts and bolts of intervention? And, most critically, what are the key elements of school turnaround?
Some 22 states have passed "academic bankruptcy" laws, allowing state intervention along a continuum that ranges from warnings to temporary leadership replacement to takeover, which implies governance change. (2) But because the issues involved are so complex, consensus is emerging that the basis for academic intervention must be imbedded in sophisticated accountability systems, incorporating appropriate assessments and incentives.
At the hub of the effort should be standards, or clearly spelled out expectations for academic content and performance by subject and grade level. But determining acceptable performance levels requires wrestling with questions whose nature is epistemological. How good is good enough? How bad is bad enough? What is fair? As researcher David Cohen and others point out, these are all issues on which Americans deeply disagree. (3) Standards setting has been contentious in virtually every state that’s attempted it. Decisions rest on finding consensus, with values being the centerpiece of the debate. And it must be acknowledged that the real action, on standards-setting and support for high performance, remains at the local level.
Policy Framework
Beyond a set of agreed-upon standards, the effort requires unprecedented clarity spelled out in a detailed set of legislative statutes on the following:
- Measures. States must determine what yardstick(s) will be used to measure each school’s performance under the standards. Questions states need to grapple with here include: Will the indicators of performance be qualitative (taking school culture and functioning into account) and/or quantitative? Will the accountability system use one measure or multiple measures? Will the assessment system employ criterion- or norm-referenced tests— or both? At what levels? Time intervals? On the technical front, how many student performance levels (e.g., "distinguished," "proficient," "novice") will figure into a school’s performance index? Will non-cognitive measures (dropout, attendance, retention rates) be part of the index? What will be the state’s overall performance target in what time period? (e.g., 90% students proficient in 10 years?) What will be the units of incremental progress? (one year? two years?) How will each school calculate its own index (score)? Will the state disaggregate school data by group, e.g., ethnic or minority or limited-English proficient?
(Some California districts are discovering that some schools do well in the aggregate but are failing a given minority group. Disaggregated data can be a safeguard, ensuring that those students don’t fall through the cracks. It can also help a low-performing school focus its resources on students who need help most, thus also boosting progress toward growth targets.)
- Rewards and sanctions. States need to spell out incentives/consequences for performance. Doing so requires asking: What are fair and appropriate rewards for high performance and/or sanctions for unacceptably low? Will rewards or sanctions be determined on the basis of absolute performance (school scored better than others) or on growth toward performance targets (each school competes against its own past record)— or both? (Implicit here is the thorny issue of whether each school’s index should be weighted for socio-economic status [SES]. As assessment expert Stanley Rabinowitz points out, the pros include a perception of greater fairness; cons are greater complexity and public confusion as well as a possible message that certain groups cannot be expected to perform at high levels. Kentucky, which focuses on progress under biennial growth targets and does not weight for SES, has found that demographics do not predict level of success, measured in terms of progress; rewarded and sanctioned schools span the full socio-economic spectrum.)
- What are the school performance categories (e.g., "successful," "improving," "in decline")? What level of state resources will support rewards and/or interventions (assistance)? Will incentives — positive and negative — exist only for school staffs or also for students? (Though more than half the states say they provide rewards or sanctions for schools or districts based on test performance, only 14 plan to promote or graduate students based on whether they have met state standards. (4) Yet Tom Boysen, former state commissioner in Kentucky, insists that "before we get more sophisticated about rewarding [or sanctioning] adults, we need to get the attention of the kids. They need to know that their performance under the standards will have big consequences in terms of getting into universities and where they’re placed there, and that employers will pay attention as well.")
- Triggers for Sanctions. At what level of performance will a school be labeled "low," i.e., at what exact point will some form of intervention (assistance) begin? If intervention will occur in phases, what criteria will define eligibility for phases 1, 2 or 3? What exactly will the intervention (assistance) be at each phase? What outcomes will be expected of the school, over what time period? How long a time will a school have to improve and, thus, end its probation or avert relegation to a more state-involved phase? By what criteria will a school be judged no longer "low performing"? Exactly what would trigger a shift from state vigilance to actual takeover? If takeover occurs, what exact criteria will determine when (and how) the state will leave?
A number of cross-cutting issues must be considered in building the system, including:
- Fairness. Apart from clarity, the most critical factor for credibility and success is fairness. If the system is not perceived as fair, it will fail. Foursquare are questions about whether all students have equitable opportunities to learn what they’re expected to know and to do well on the tests designed to measure that learning. Critical, then, is getting input from as many stakeholders as possible. Those needing a voice in the process include teachers, administrators, business and industry representatives, unions, parents and school board members.
- Focus. Builders of these systems must maintain a focus on the forest while in the thicket of trees. Throughout, they need to remember that this elaborate machinery is only a means to an end. The goal is supporting and enhancing student learning. Everything else is just the "how."
- Coherence. If the assessment system is not aligned with standards, the system will fail. Even more essential is that whatever measures are used must measure what you value — again underscoring the need for public input. If you value what students know and can do but measure what they have memorized, the assessment will drive the system away from the outcomes you want. Fairness becomes impossible.
- Understandability. A major challenge is balancing fairness — which necessarily creates complexity — with the need to have a system that is readily understood by the educators it affects and the public it seeks to serve. Related to this is the state’s need to communicate clearly with critical players such as the media what the new system aims to accomplish and how.
- Capacity building. Threats or sanctions will not increase student achievement if the system discourages, rather than motivates and supports, teachers and administrators. Their efforts need to be bolstered by the professional development, resource and assistance they need to do the job.
- Legal defensibility. Courts are specific about what you can and cannot do with tests. Legal challenges are most likely to surface when assessments carry high stakes — for example, when students must pass a particular test to graduate. Other bases for legal challenges include tests being used for purposes other than those for which they were intended, tests measuring knowledge or skills that students have not had the opportunity to learn, or tests producing an adverse impact on historically disadvantaged groups. (5)
Sanctions or interventions generally begin when a school or district fails to meet its clearly spelled-out improvement targets within an agreed-upon time period. The first phase is likely to require that the school staff, working with parents, community members and the district, do a comprehensive needs assessment and create an action plan wherein quantifiable learning goals and time frames are clearly delineated. Fairness dictates that schools at this phase must have at least as much leeway to be creative and take risks as an intervention team would have. (Schools often don’t realize how much autonomy they have. As Oregon moved to enact an accountability system, 18 schools requested waivers; 17 of those didn’t need them for what they wanted to do.)
If the school fails to improve sufficiently, it would go to the next phase, which is likely to mean assignment of a distinguished educator (DE) and/or other outside expert or team who would facilitate a transformation planning process. If the school still failed to improve within a designated amount of time, much more aggressive intervention would occur. Under California’s proposed system, for example, once a state reaches this crisis point the State Board of Education would step in and determine whether to continue second-phase activities, reassign or transfer students or staff, reallocate resources or close the school. (6)
Specifics differ from state to state, depending on the philosophy that underpins them. For example, some states have withheld money from schools if performance remains low; others withhold accreditation, which may lead to cut-off of funds. Espousing an opposite psychology, North Carolina has intervened by enforcing teacher testing, then offering scholarships— in the form of paid time to learn and substitutes in their classrooms— to teachers who need help. Other ways of tying incentives to professional development include giving a wage credit when teachers take courses, but only if those courses are in keeping with a school’s or district’s goals.
As states struggle to find best ways to implement academic intervention, most look to Kentucky, whose five-year-old accountability system is often cited as a prototype. Through its DE program, the state has now intervened with 240 of its 1400 schools, and the DE program itself— with its by-products and growing voice in policy— has become a catalyst for systemic change. Robert Lumsden, Kentucky’s associate commissioner of education, offers the following advice, based on that program’s experience:
- Throw all guns at the new program. You’ll need success and positive examples fast.
- Expand and market successes. The legislature and public want to see spin-offs from their investment. In Kentucky, the DE training evolved into a leadership academy; the curriculum alignment approach required for schools in the DE program is being adopted by other schools as well.
- Examine and re-examine intervention strategies. Kentucky commissioned a study of its strategies even before getting results; evaluators gave the approach high marks.
- Use a common set of strategies across all schools so you can talk about the program coherently. But avoid inadvertently conveying a message that solutions are the same at each school. One strategy used across schools in
Kentucky is curriculum alignment. But DEs have much latitude to tailor the "how" and "when" of the process to the readiness of each school’s staff, a point that needed "marketing" to clarify early misunderstandings.
- Keep the management system simple. Overall management in Kentucky has three categories, with one state-level person in charge of each: recruitment, logistics (organizational issues), and "cut to the chase"— are we staying on course (with the conceptual model) and focusing on results (monitoring data)? Implementation is led by the DE teams so that decisions are grounded in the intervention, not the bureaucracy.
- Focus on measuring what’s important. Kentucky insists that no paperwork be required from a DE that doesn’t clearly support meeting the school’s improvement goal.
- Identify and fix structural problems. Implementing the DE program has prompted Kentucky to remove obstacles to its success. For example, the planning process once required separate applications for federal and state discretionary money; now there’s one application— a change that required organizational shifts at the state department. Moreover, the state and DE leaders worked together over a two-year period to strengthen high school graduation requirements after DEs pointed out that some students were not getting the opportunity to take the courses they needed to perform well on state exams.
- Expect that the costs will be high. Kentucky has found it costs $200,000-$250,000 over two years to turn a school around to the point of meeting its improvement goal. They’re trying to get that figure down to $100,000. Yet Lumsden relates a growing belief that success depends on "a human, labor-intensive program with lots of one-on-one contact" between the change agent and the school. That requires sufficient allocation. He loops back to his first point — show results. "Results become the negotiating tool."
At the School: Key Elements of Turnaround
The true challenge of devising the mechanics of intervention is to make sure approaches used accommodate and incorporate the human factors. What do research and experience reveal makes sense, psychologically and motivationally? What baseline kinds of support do school staffs need in order to change? Important elements for success include:
Legitimacy, Reciprocity and Trust
State intervention systems assume that the threat of sanctions will motivate educators to improve practice and, thus, student performance. But organizational research suggests that such a threat can engender one of two reactions: 1) the desired response, i.e., the group becomes more cohesive, leadership develops and improvement results, or 2) the opposite — there is less cohesiveness and more divisiveness and leadership falls by the wayside. Researcher Jennifer O’Day, who has been documenting reconstitution (see box p.10), points out that the prime ingredient for ensuring the desired reaction is whether or not the people involved believe that the criteria and process are legitimate and fair. (7)
- Legitimate criteria. Do people understand the outcomes being sought and the criteria being set? Do they respect the criteria, i.e., does the system measure something important in terms of student performance? Are the criteria multi-faceted, or will people say, "If this test is the sole criterion, it’s not fair"? In Chicago, when an accountability system was adopted, some schools that had been improving were rated very low. What message did this send?
- Fair process. Do school staffs think the process of identifying schools and intervening with them is fair? Do they trust that purposive action, rather than sheer luck, will make a difference in whether or not more serious sanctions occur? Or do they believe that nothing they do will matter, so they might as well look for another job? For a time in San Francisco, six to seven schools a year were being placed on probation. However, because a consent decree called for at least three to be reconstituted, schools identified often feared that the odds were against them. That could serve as a de-motivator.
- Respect. Teachers need to feel sure that despite a focus on students, adults also matter, says O’Day. Even in schools with entirely new staffs, recruited on the basis of their willingness to work long hours, be a team and be held accountable for student learning, tensions exist simply because schools are communities of young people and adults. If a student is repeatedly, violently disruptive, for example, a stance of "We will keep him in school no matter what" may need to shift to "Maybe at this time this is not the place for this child." "The bottom line has to do with the place people work and the kind of support they feel," says O’Day. "This kind of relationship, this trust, is as much of an incentive as a monetary reward."
- Capacity and support. Does the school staff believe they have the capacity to alter the conditions in the school? Does the system balance threat and assistance by providing needed resources for professional and organizational development?
School-Level Leadership
A critical element of the capacity to change is leadership. A strong leader inspires people to believe in themselves, creating immediate optimism, a sense of common mission, and a powerful "can do" attitude. Summed up by New Jersey State Commissioner Leo Klagholtz, "If you believe in what you’re working on, that belief is almost more important than the value of the strategy."
The urgency to bring in, develop and support strong principals is repeatedly underscored by experience and research. A study done on takeovers in New Jersey by Arthur Andersen noted that whether or not schools progress is less related to factors such as poverty or mobility than to school leadership. A critical piece, Leo Klagholtz corroborates, is changing the tenor of leadership, school by school. "Success hinges on the ability of the principal to translate the school’s strategic plan into action by the faculty and community."
Robert Lumsden cautions that Kentucky’s experience indicates that in at least 60 percent of interventions, you can change the tenor of leadership by supporting, not replacing, the existing principal. "You focus on improvement, build on that person’s strengths." If the principal doesn’t improve, the situation will self-correct once a DE is assigned to a school, he says, because "new community awareness of how poor he is will drive him out." Lumsden also emphasizes the importance of leadership stability after the intervention team leaves. For the past year, Kentucky has been tracking 30 turned-around schools to see what happens after the DE leaves. "Five have slipped, and four of those have had changes in principals."
Instructional Quality
Teaching is the heart of the matter in whether students learn. If the teaching staff does not have the knowledge and skills to bring the kids they teach to high standards of achievement, then no manner of threat, intervention or leadership can turn around a low-performing school.
The challenge is to support teachers in learning what they need to know and changing their attitudes and practices. Much research shows that the assistance teachers need is not one-shot workshops or "training" but professional development that is intensive, ongoing and linked to practice. In the inner city, home of many of the nation’s low-performing schools, the need for this kind of professional development is especially acute. These schools have the hardest time attracting qualified, experienced teachers. Yet the children in them are the most likely to be minority, poor or affected by other life circumstances that make them most in need of sophisticated teaching practices.
Resources
There is wide agreement that simply throwing money at the problem isn’t going to solve it. But without sufficient funding, schools can’t harness the human and intellectual resources essential to progress. U.C. Berkeley professor Pedro Noguera has praised the San Francisco Unified School District for being the only large, inner-city district in the country that has seen test scores, graduation rates and a host of other achievement indicators rise consistently over the past five years. (13) District policy changes have been aimed at elevating academic standards for all students. Noguera sees money as a clear factor in this success, and San Francisco has more of it than other California big city districts (though less than counterparts in the East). But he says what seems key is the coupling of resources with leadership and commitment. Superintendent Waldemar Rojas has focused "laser -like" on strategies for improving student achievement, Noguera notes, especially emphasizing the needs of the poorest and least prepared students (see also "Reconstitution," p. 10).
Academic intervention strategies entail many unanswered questions. Worries remain about using test scores as the sole determinant of which schools are labeled low performing, as is happening in a number of places. Debate also continues over how to determine reasonable timelines for improved performance, especially for student achievement. Most fundamentally, how best to structure and direct resources to address the complex of leadership, staff, organizational and community development issues at play in a failing school or district remains more art than science.
Clearly, one great challenge is combining the big stick and the helping hand and pooling talent to push for results. States need to avoid sending a "gotcha" message that will offend professionalism, trigger defensiveness and, thus, subvert their own goal. As Jennifer O’Day notes, the real key to success appears to be reciprocity—between those designing the system and those in the schools. States and districts can and should expect schools to function at their best and serve students well. But schools also have the right to expect that they will be given sufficient resources and support to do the job.
—Joan McRobbie
With input from other WestEd staff members, including Lisa Carlos, Stanley Rabinowitz and Paul Hood.
1. Lynn Olson, "The Push for Accountability Gathers Steam," Education Week, February 11, 1998.
2. Amy Berk Anderson and Anne C. Lewis, Academic Bankruptcy (Policy Brief), Denver, CO: Education Commission of the States, March 1997.
3. David K. Cohen, "Rewarding Teachers for Student Performance," in Susan H. Fuhrman and Jennifer A. O’Day (editors), Rewards and Reform: Creating Educational Incentives That Work, San Francisco, CA: Jossey-Bass Publishers, 1996.
4. Lynn Olson, "Keeping Tabs on Quality," Quality Counts: A Report Card on the Condition of Public Education in the 50 States, (Education Week Special Report), 1997.
5. Robert L. Linn and Joan L. Herman, A Policymaker’s Guide to Standards-Led Assessment, Los Angeles, CA: National Center for Research on Evaluation, Standards and Student Testing and Denver, CO: Education Commission of the States, February 1997.
6. Steering by Results: A High-Stakes Rewards and Interventions Program for California Schools and Students, Sacramento, CA: Report of the California Rewards and Interventions Advisory Committee, 1997.
7. Jennifer A. O’Day of the University of Wisconsin-Madison is associate director of the Pew Forum on Education Reform, a continuing body of education leaders, researchers, practitioners, and advocates who share a common interest in improving U.S. schools. This document draws from her presentation to a gathering of chief state school officers and others in Fall 1997 and from a February 1998 interview.
8. Linn and Herman, A Policymaker’s Guide to Standards-Led Assessment.
A New Accountability System for California Schools, California Department of Education and NorthernCalifornia Comprehensive Assistance Center (WestEd), 1997.
9. John A. Nunnery of Johns Hopkins University as quoted by Lynn Olson, "Teachers Need Nuts, Bolts of Reforms, Experts Say," Education Week, April 30, 1998.
10. A Policymaker’s Guide to Education Reform Networks, Denver, CO: Education Commission of the States, 1997.
Margaret C. Wang, Geneva D. Haertel and Herbert J. Walbert, What Do We Know: Widely Implemented School Improvement Programs, Philadelphia, PA: Laboratory for Student Success, 1997.
Achieving Nationwide School Improvement Through Widespread Use of Effective Programs and Practices, Crespar Research and Development Report, Baltimore, MD: Center for Research on the Education of Students Placed At Risk, September 1997.
11. The Daily Report Card, Washington, D.C.: Education Policy Network, Inc., National Goals Panel, March 18,1998.
12. O’Day, presentation and interview.
13. Pedro A. Noguera, "No Excuses Accepted: How One Urban District Succeeds Where Others Fail," Education Week, November 5, 1997.
This paper is an outgrowth of a Fall 1997 meeting co-sponsored by the State Education Improvement Partnership at the Council of Chief State School Officers, the Nevada Department of Education and WestEd
Written by Joan McRobbie, with input from other WestEd staff members, including Lisa Carlos, Stanley Rabinowitz and Paul Hood.
Hard copies of this paper are available for $4 each at WestEd. Contact Danny Torres at 415.615.3144 and ask for "Can State Intervention Spur Academic Turnaround?" published Spring 1998
For more information contact:
Joan McRobbie
WestEd
730 Harrison St.
San Francisco Ca 94107Voice: (415) 565-3069
Email: jmcrobb@WestEd.org