This Annotated Bibliography was prepared by Leah Altemeier, Lisa Carlos, Don Klein, and Joan McRobbie of the Policy Support & Studies Program at WestEd. Its purpose is to introduce readers to selected articles that raise some of the complex issues involved in designing accountability and incentive structures. It was funded in part by the U.S. Department of Education, Office of Educational Research and Improvement.

Contents:

Abelmann, C.H., & Kenyon, S.B. (1996, April 12). Distractions from teaching and learning: Lessons from Kentucky's use of rewards. Paper presented at the meeting of the American Educational Research Association Annual Conference, New York, NY.

Anderson, A.B., & Lewis, A.C. (1997, March). Academic bankruptcy. Policy Brief from Education Commission of the States (ECS), Denver, CO.

Bryk, Anthony, Shipps, Dorothy, Hill, Paul, & Lake, Robin, et. al. (1997). Decentralization in practice: Towards a system of schools (draft). Paper presented at the meeting of the American Educational Research Association Annual Conference, Chicago, IL.

Clotfelter, Charles, T., & Ladd, Helen F. (1996). Recognizing and rewarding success in public schools. Holding Schools Accountable. Washington, D.C.: The Brookings Institute.

Cohen, David K. (1996). Rewarding teachers for student performance. Rewards and Reform: Creating Educational Incentives That Work. San Francisco: Jossey-Bass.

Elmore, Richard F., Abelmann, Charles H., & Fuhrman, Susan H. (1996). The new accountability in state education reform: From process to performance. Holding Schools Accountable. Washington, D.C.: The Brookings Institute.

Heneman, Herbert. (1997, March 24-28). Assessment of the motivational reactions of teachers to a school-based performance awards program. Paper presented at the American Educational Research Association Annual Conference, Chicago, IL.

Kelley, Carolyn. (1997, Spring). Teacher compensation organization. Educational Evaluation and Policy Analysis, 19(1), 15-28.

Kelly, C. (1997, March). The Kentucky school-based performance award program: School-level effects. Paper presented at the meeting of the American Educational Research Association Annual Conference, Chicago, IL.

Ladd, Helen F. (1996). Introduction. Holding Schools Accountable. Washington, D.C.: The Brookings Institute.

Newmann, F.M., King, M.B., & Rigdon, M. (1997, Spring). Accountability and school performance: Implications from restructuring schools. Harvard Educational Review 67(1), 41-74.

Odden, A., Heneman, H., Wakelyn, D.J., & Protsik, J. (1996, October). School-based performance award designs: A case study. Paper presented at the meeting of the American Educational Research Association Annual Conference, Chicago, IL.

SouthEastern Regional Vision for Education (SERVE). (1994, September).A new framework for state accountability systems. Special Report. Greensboro, NC.

Suarez, Tanya M. et. al. (1991, June). The use of sanctions with low performing school districts. Chapel Hill, NC.

Abelmann, C.H., & Kenyon, S.B. (1996, April 12). "Distractions from teaching and learning: Lessons from Kentucky's use of rewards." Paper presented at the meeting of the American Educational Research Association Annual Conference, New York, NY.

Abelmann and Kenyon begin with an historical overview, explaining that states have shifted from school improvement policies that emphasize inputs or process regulations to those policies that provide monetary incentives. According to the authors, states and districts developing such plans have typically asked: 1) what cognitive and non-cognitive indicators should be used to index performance; 2) should such an index correct for racial and SES bias; and 3) should collective or individual incentives be used? The authors go on to discuss differences among state rewards schemes, such as judging a school's performance against a relative standard or its own benchmark and basing teacher merit pay on subjective supervisor evaluations of teacher performance or on objective criteria - typically standardized student test scores.

The remainder of the article analyzes data derived from four school-level case studies, news articles, and state education department surveys. This analysis is organized around five assumptions.

The first assumption-that teachers understand and are motivated by the concept that they will be rewarded for good performance-was proven untrue. In these four cases, teachers did not view themselves as participating in "an even exchange." Moreover, "there is little indication that rewards acted as an incentive at all" and "fear of sanctions" or fear of becoming a "school in crisis" was a stronger motivator than incentives.

A second assumption-that intrinsic rewards (e.g. quality of working environment) are more important than extrinsic rewards (e.g. financial incentives) in motivating people, was found to be true. In these cases, extrinsic awards were less important or conflicted with the intrinsic value teachers felt about their professions.

Third, the authors question the strategy of spending resources on rewards as opposed to building teacher capacity (e.g. knowledge and skills about best practice). In lieu of such capacity, "each school takes its best guess at practices that might assure success…"

Fourth, the authors ask whether incentives should solely be aimed at teachers, since teachers are not the only determinant of student performance. The authors point out, for example, that students are not recipients of cash incentives, yet they too need to be motivated.

Finally, many teachers were skeptical that cash awards would ever materialize. Although the authors concede that performance in some schools has increased, this article raises issues about whether or not such an increase is directly related to cash incentives.

The authors list six lessons learned from these case studies of four schools: 1) teachers should be involved in the development of an incentive system, so that they can signal to policy makers whether they are motivated, for example, by intrinsic over extrinsic rewards; 2) rewards should be linked to individual student progress, not cohort comparisons; 3) inequities among schools regarding their capacity and access to the knowledge necessary to enact reforms that improve performance must be addressed; 4) the reward process should not be left up to school level staff and use of such funds should be clearly tied to school improvement, not individual salaries; 5) incentives should be tied to clear goals within a reasonable time frame; and 6) to take the reform seriously, teachers need assurance in the promises made and the long-term stability of the effort.

< back to Contents >

Anderson, A.B., and Lewis, A.C. (1997, March). "Academic Bankruptcy." Policy Brief from Education Commission of the States (ECS). 707 17^th Street, Suite 2700. Denver, CO 80202. (303) 299-3600.

Twenty-two states currently have state takeover laws for schools experiencing "academic bankruptcy", or a consistent failure to meet state education performance standards. This policy brief provides a summary of the provisions for the different states and discusses academic bankruptcy in practice from three states' experiences. For almost all states, interventions usually occur in stages, with districts first being identified by the state as having a number of low-performing schools, with a period of time (usually up to one year) for the school's district to develop an improvement plan. If the district is still underperforming, it receives state intervention, which can often involve hiring a consultant to conduct a comprehensive audit and make recommendations for improvement to the state board of education. If after another period of time (usually one year), the district has not improved, some form of takeover may occur. Contained in this brief is an outline of the levels of authority at which each state is permitted to intervene; for example, some states have the power to send technical assistance teams for audit purposes, appoint interim administrators, remove the superintendent, staff and/or school board, dissolve districts/schools, withhold funding from underperforming districts, and/or notify the public.

Following is a section listing arguments used both for and against aspects of the academic bankruptcy issue, such as the efficacy of state governance, the impact on teaching and learning, the use of rewards and sanctions, and the effects on school staff and community members attitudes.

Three case studies elucidate some practical experiences. Baltimore City Public Schools failed to improve even after the mayor appointed school board members and the superintendent, and after the City Council directed the school district budget. It was the added element of accountability for the school board members, chief executive, academic, and financial officers to improve academic performance that began to produce results. The second lesson for Baltimore was that partnerships are more effective than top-down strategies. Ohio legislation has recently stated clear state expectations, used a formal calculation to determine "satisfactory progress", and provided for sensitivity to the uniqueness of each district. New Jersey is operating three urban school districts, assigning an urban specialist, providing special coordinators and services for programs, holding monthly meetings between the state department of education and district leadership, providing an Urban Coordinating Council that handles the administrative red tape, and giving funding for districts to develop improvement plans. Since the state has seen satisfactory improvement in all districts over the seven-year period, it is developing guidelines for continued progress and safeguards for the near future when districts re-assume local control.

< back to Contents >

Bryk, Anthony, Dorothy Shipps, Paul Hill, and Robin Lake et al. 1997. "Decentralization in Practice: Toward a System of Schools" (Draft). Paper presented at the meeting of the American Educational Research Association Annual Conference, Chicago, IL.

This paper presents an overview and findings of a 2 ½-year study of school reform efforts in several cities that have tried to decentralize school operations. The study focused on six large cities that have, in different ways, sought to increase local schools' freedom of action and emphasize greater accountability for results. The cities were Charlotte-Mecklenburg, Chicago, Cincinnati, Denver, Los Angeles and Seattle. In sum, the authors found that the pace of reform in these cities remained slower than expected. They found that loosening controls on schools is not in itself a sufficient condition for improvement. Left to their own devices, some individual schools have moved forward, but many more continue to struggle and experience much less success. Primarily, none of the decentralization efforts studied sufficiently attended to the external assistance that teachers, principals, and parents need to meaningfully redress the quality of instruction and other services to children. None of the districts fully considered how its extant system of incentives and sanctions may need to be reframed to aggressively stimulate improvements.

As one component of the larger study, the authors did examine the efforts to implement new systems of school-based accountability. The authors found that only Charlotte-Mecklenburg included accountability designs in their original reform plans, while Chicago and Cincinnati added them only after reforms had begun, and in Denver, Los Angeles and Seattle such plans were either unclear or not present. The authors conclude by noting that decentralization opens up the possibility of four different types of accountability: hierarchical district accountability (schools and students must meet learning requirements), customer accountability (schools must attract students), professional accountability (schools must win accreditation and maintain respect of professionals), and political accountability (school leaders must answer local site councils). According to the authors, no locality has yet committed the necessary resources nor established the governance arrangements to implement a decentralization program fully integrated with a comprehensive accountability system.

< back to Contents >

Clotfelter, Charles T., and Helen F. Ladd. 1996. "Recognizing and Rewarding Success in Public Schools." Holding Schools Accountable. Washington, D.C.: The Brookings Institution.

This article examines in detail two initiatives that combine a focus on student outcomes with financial incentives for schools, commonly referred to as recognition and reward programs. In particular, they compare the experiences of the state of South Carolina since it adopted its incentive award program in 1984, and those of the Dallas Independent School District, which adopted a similar program in 1990, regarded by the authors as the most complete and sophisticated accountability and incentive program in the country. The main questions the authors address are: how to pick winning schools; whether or not to adjust for socioeconomic differences among schools; what sorts of incentives are generated; and whether such programs are likely to increase student learning. Their main findings are: such programs will not meet the fairness test unless test scores are adjusted for the students' socioeconomic status; while attempts to make these adjustments in SC and Dallas have increased fairness they have also resulted in incredibly complex methodologies that render the process confusing to participants and the public; and rewards that have been large enough to change behavior have also engendered less than desirable results, such as cheating and data manipulation.

< back to Contents >

Cohen, David K. 1996. "Rewarding Teachers for Student Performance." Rewards and Reform: Creating Educational Incentives That Work. San Francisco: Jossey-Bass

Cohen outlines a complex analytical framework that includes three key themes. The first is the lack of reliable professional or social science knowledge about the effects of such schemes. Cohen argues that "we know enough to rule out simplistic and punitive schemes-though that may not be sufficient to keep them from being adopted-but not enough to inform choices among more complex and sophisticated alternatives." Another tension is that rewards alone do not fix the complex problems facing the worst schools. Despite arguments to the contrary, resources are needed to improve the capacity of schools. Finally, Cohen argues that educators and the public will need clarity about the measures of performance, criteria of success, and fairness associated with a rewards scheme. However, striving for such clarity can often exacerbate fundamental disagreements, which-in turn-raise other complex issues.

Several other questions are explored. One involves how best to measure student performance. Which standardized or performance-based tests are more likely to drive student learning and school performance in the right direction? tests? Each has its merits and drawbacks. A second question concerns the criteria used to determine school success. Should single or multiple criteria be used? What compensatory adjustments, if any, should be made for non-school influences on student performance, such as parental education and income level? Another issue is how to improve "professionalism" or the professional capability of a school to respond constructively to a rewards scheme. The chapter continues exploring these and other issues and concludes by reiterating that additional research is needed before some of these important questions can be answered.

< back to Contents >

Elmore, Richard F., Charles H. Abelmann, Susan H. Fuhrman. 1996. "The New Accountability in State Education Reform: From Process to Performance." Holding Schools Accountable. Washington, D.C.: The Brookings Institution:

In this article, Elmore, Abelmann and Fuhrman describe an emerging model of state and local governance and accountability: a "model of steering by results"-using rewards, sanctions and assistance to compel higher levels of school performance. Few states have a performance-based system fully operating, but at least 80 percent of 50 states surveyed in 1993 indicated they were engaged in developing, piloting or implementing some form of this new approach to accountability. Several states, for example, have experimented with financial incentives, including Indiana, Kentucky, North Carolina, South Carolina, and Texas. Meanwhile, other states developed interventions for severely troubled districts and schools that lost accreditation. Interventions include the appointment of on-site monitors, followed by state takeover or removal of local governance. In Maryland, sanctions include privatization and reconstitution of schools' staff. Other states focus on program improvement through intensive on-site inspection, which includes classroom observations and lengthy feedback, such as the Program Quality Review process found in California.

Based on follow-up interviews in 1994, the authors selected Kentucky and Mississippi as the two states farthest along in the development of performance-based systems and the focus of more in-depth analysis. They represent opposite ends of the continuum: KY is viewed as a higher cost, more complex version of performance-based accountability, and MS as a lower cost, basic version. Kentucky, unlike Mississippi, is a higher stakes model, in that the financial incentives and sanctions are far more significant. The Mississippi system provides technical assistance for low level districts and limited incentives, in the form of deregulation, such as exemption from staff development day requirements, for high performance districts. In this state, the unit of accountability is the district, as determined by a district accreditation process. The KY system, meanwhile, is more comprehensive and holds schools and districts accountable as the unit of analysis.

Schools in Kentucky are accountable for meeting performance standards, as measured by a newly created performance-based assessment system, in eight domain areas: reading, writing, mathematics, science, social studies, arts and humanities, practical living and vocational studies. In 1995-96, MS state switched to a new performance-based assessment system which tests all students in grades 4-9 (but most of CPRE's analysis is based on their prior standardized testing system). In MS, performance is based on the notion that performance is relative. In KY the message conveyed is that performance is an absolute phenomenon, meaning all schools could potentially have a significant number of students performing below the expected standard in all domains.

The two states have faced a common set of implementation issues. One is the extent to which these systems focus on schools that fall at either end of the performance scale, thereby paying virtually no attention to schools in the middle. Issues of the fairness for holding districts and schools that serve differing numbers of poor and transient students have also been raised. But the data in KY indicate that the distribution of awards sponsored the entire SES spectrum. Another important set of issues has to do with state department capacity to assist low performing schools. Despite MS efforts to hire special consultants to provide support, such support has been considered "short-lived." In KY, the department went through an exhaustive search process for "distinguished educators" to assist poor performing schools, but the General Assembly cut funds for this program by 40% while simultaneously cutting back the department's staff. The article details the political pressures imposed by different constituents, fiscal constraints, lack of public understanding and the ongoing ambivalence towards shifting to a deregulated system.

The article ends with five main challenges for states as they move toward new accountability: 1) making accountability systems understandable and defensible to policymakers, educators, parents, and students; 2) resolving issues of fairness in the design and implementation of accountability systems; 3) focusing incentives for improvement; 4) developing state capacity to implement and maintain accountability systems; and 5) creating a stable political environment.

< back to Contents >

Heneman, Herbert (1997, March 24-28). "Assessment of the Motivational Reactions of Teachers to a School-Based Performance Award Program." Paper presented at the American Educational Research Association Annual conference, Chicago, Illinois.

This case study describes the design and operation of a school-based performance award program used by the Charlotte-Mecklenburg schools and its impact on teachers' motivation. The author, Herbert Heneman, based the study on "expectancy theory"-- a theory explaining employee motivation to perform. That theory posits that employees operate according to several expectations: the probability that high effort will lead to high performance; high performance will lead to various outcomes (bonuses, etc.); and that these outcomes are highly desirable.

As part of this study, twelve bonus recipient and non-recipient schools were chosen for inclusion, selected by districts with several criteria preset by the researchers. Interview protocols were followed with individual principals and with groups of between 3 and 8 teachers. Major themes and issues were identified according to the researchers' theoretical model.

Qualitative data on teacher motivation found that, in accordance with findings from the Kentucky program, teachers viewed student achievement goals as understandable, appropriate, specific, focused, and attainable; moreover, they agreed that it was appropriate to be held accountable for these goals. Discussion about teaching "enablers", or facilitative factors such as resources, the principal, professional development, and school culture, were lengthy. Although they felt the school had adequate resources, teachers wanted more professional development and information about "best practices". Bonus amounts were criticized as taxes typically deducted more than $400 from a $1000 award; nevertheless, teachers interpreted receipt of the bonuses as a source of pride and recognition. This source of professional pride was an important motivating factor for teachers. They also recognized the importance of enablers in helping them accomplish their goals.

These results were similar to those found with the Kentucky program (Kelley, 1997), with several differences. Measures of student achievement were more performance-based (e.g., results based on student writing) and complex in Kentucky than in Charlotte-Mecklenberg schools. Teachers in Kentucky felt their bonuses were large enough, and the authors report that for bonuses to have maximum motivational value, they should be at least 5% of base salary. In contrast with Charlotte-Mecklenberg teachers, Kentucky teachers voted on how to divide the bonus award for their school among teachers and staff, creating some internal conflict. Finally, Kentucky teachers were very aware of negative sanctions their schools could experience, and many were more motivated to perform better as a way to avoid the negative sanctions than as a way to achieve positive rewards. Charlotte-Mecklenberg teachers did not feel this pressure.

< back to Contents >

Kelley, Carolyn (1997, Spring). "Teacher Compensation and Organization." Educational Evaluation and Policy Analysis, 19 (1), 15-28.

The author, Carolyn Kelley, provides an historical overview of the evolving models for teacher compensation. Her basic point is that teacher compensation is one aspect of educational organization that has not changed in accordance with recent systemic reform efforts. Outlining 20^th Century school organization changes that have resulted in new roles, skills, and knowledge requirements for teachers, as well as performance expectations, Kelley highlights how teacher compensation should be changed to better complement these changes. Different compensation schemes are discussed, such as pay raises based on seniority, skills or competencies, and performance outcomes. Though viewed as equitable, she argues that the single salary schedule rewards teachers on the basis of longevity and education alone, resulting in a teacher compensation method that no longer supports current reforms in organizational structures and human resource needs. From a systems perspective, this lack of change in teacher compensation in turn impedes organizational change, explains Kelley.

The article describes the history of education reform using four conceptual models of school organization, each with its own implications for teacher compensation. From the 1900s through the 1950s, the Scientific Model of organization prevailed in schools, in which hierarchy and formal authority were valued. Teachers' roles were limited to structured curriculum implementation and rule-following. Under this model teachers received compensation for membership and seniority.

In the 1970s, the Effective Schools model valued equity for student populations over results-driven goals. Teachers' roles under this model were to teach basic skills, develop generic teaching skills, and assist in school improvement planning. Implications for compensation increases include seniority, plus rewarding professional development in teaching skills.

The 1980s brought a shift of focus from basic skills and at-risk populations to an enhanced curriculum. The Content-Driven model sought high levels of competency and encouraged teacher training in specialized content areas. Teacher compensation that fits this model would include pay raises for increased skills or competencies, and incentives for school-level behaviors, with the skills component as most important.

The fourth model of organizational change is the High Standards/High Involvement model, which increases the demand for improved student achievement outcomes rather than just changes to the curriculum. Teacher leadership becomes important, further broadening teachers' roles. Compensation implications include pay raises for the skill component, and, for the first time, bonuses for student outcomes.

In conclusion, Kelley briefly mentions the Kentucky and Charlotte-Mecklenburg programs as examples of schools that link performance awards to broader educational goals. Kentucky's program rewards schools for improving student performance along multidimensional state criteria. In the Charlotte-Mecklenburg school district, teachers are rewarded for several types of improvements, including increased student enrollment in higher-level classes and standardized test scores.

In line with private sector findings, these two programs have found that bonuses for improved performance do not necessarily risk large amounts of teacher salary. Citing private sector research, Kelley states that small rewards can serve as effective incentives, as long as the programs are designed with the input of affected employees. This is a key point, she explains, since teachers are in the best position to understand obstacles to performance improvement. Although the impetus for compensation reform may be initiated from management, Kelley states that successful districts have used participatory processes from teacher unions and community members.

< back to Contents >

Kelley, C. (1997, March). "The Kentucky School-Based Performance Award Program: School-Level Effects." Paper presented at the meeting of the American Educational Research Association Annual Conference, Chicago, IL.

Kelley's study builds upon previous research on the effects of school-based performance award programs in Kentucky, and examines the differences between high-and low-performing schools.

The program consists of using the state's authentic assessment instrument (KIRIS) every two years, administered in grades 4-5, 7-8, 11-12, and includes portfolios, performance events, open-response and multiple choice questions in seven subjects. Non-cognitive components are also included: student attendance, dropout and retention rates, and transition to adult life. Student scores fall in one of four categories: novice, apprentice, proficient, and distinguished. Performance targets are a 10% increase in performance from each school's initial score. Following initial assessment, schools can fall in one of five categories: "in crisis" (schools that drop more than five points below baseline assessment), "in decline" (five or less points drop from baseline), "improving" (achieve over baseline but fail to reach target), "successful" (meet their target), and "reward" (exceed target, and receive the award). Awards go to the entire school, with the teachers deciding how to distribute the money.

Results show that in the first biennium, 40% of schools exceeded their performance target and received performance awards. Schools that were deemed "improving" needed to submit a transformation plan to the state, outlining how they would meet goals in the future. Schools "in decline" were appointed a "distinguished educator" for one year and given funding, and also submitted a transformation plan. A distinguished educator is hired and trained by the state to help these schools meet accountability goals. Schools "in crisis" are similar to schools "in decline", but in these cases the distinguished educators have broad latitude to terminate staff and override school site council decisions.

Following the assessment results of Fall, 1996, the author selected a sample of 16 elementary, middle, and high schools, and researchers conducted interviews with principals and selected teachers. Kelley found different "enabling conditions", or organizational capacity and alignment of resources with accountability program goals, in higher success schools. Specifically, these schools shaped their curriculum to the assessment instrument and/or the state curriculum guides, and they incorporated test-taking strategies in the curriculum. Schools that received rewards both assessment periods had teachers that were professionally "in the loop" with the accountability program, and because of this, these schools could take advantage of information provided by the state as a guide to improve performance. These reward/reward schools did not necessarily have traditionally strong principals; more likely, these principals' roles were more like facilitators with their skilled and professional teaching staff.

Low-performing schools, on the other hand, did not attempt to incorporate assessment goals into their curriculum. The principal was less an authoritative figure than a nurturing one, and did not tend to exercise leadership in encouraging teachers to shape their traditional instruction to ones with proven performance. Teachers tended to view assessment results more as a reflection upon their students than upon their teaching. Principals indicated lower personal goals of achieving slight improvement of KIRIS scores rather than achievement of reward status.

After the first assessment, 53 schools were considered in decline. Of these, 31 received awards in the second cycle, and the remaining 22 were either improving or successful. Turn-around schools, or schools that did not receive rewards the first cycle, but received them in the second cycle, shared many characteristics: teachers attributed their success to the extra funding and to working with distinguished educators, reporting this type of intervention as incredibly difficult, even painful, but also as one of their most professionally rewarding experiences. In addition, both decline/reward schools hired a new principal after the first award cycle, with strong leadership and track records of success in the accountability program.

As in Kelley's initial study of reward schools (see page 11), schools in the second reward cycle did not attribute their motivation to expectations of financial reward. They indicated a fear of sanctions, positive publicity, and an interest in helping their students as their primary motivating factors. Some teachers were also motivated by a desire to maintain professional autonomy. A few teachers and more principals were fearful of losing their jobs. Rural Kentucky schools mentioned that the KERA resources and the focus on relative improvement over absolute performance, enabled them to compete with even the best Kentucky schools. Still, the reward served as a "thank you" for teachers' hard work. However, there were many conflicts over the distribution of reward money, and the author notes that because of this the Kentucky Education Association opposed the funding of bonus awards in 1997.

Other findings showed that elementary schools were found to be more successful in achieving reward status than middle and high schools. Teacher explanations for this were the organizational structure, and the flexibility of the program and staff of elementary schools. Middle schools had difficulty motivating adolescents who were preoccupied by their own developmental changes. High schools were described as large and departmentalized, which impeded collaboration. Also teachers in some high school subjects were uncomfortable with the large emphasis on writing and thus not as cooperative.

< back to Contents >

Ladd, Helen F. 1996. "Introduction." Holding Schools Accountable. Washington, D.C.: The Brookings Institution.

As an introduction to the larger book, this article provides an overview of the two sides of performance-based reform movement: accountability initiatives designed to work within existing administrative structures of schools, and accountability initiatives designed to work through parents' ability to choose. From this overview, a series of questions emerge: Who should be held accountable - teachers, schools or districts? Are administrative accountability systems desirable and sustainable? What are the crucial components of a well-designed accountability system? (For example, how do you account for schools whose kids have larger needs? How do you incorporate the call for more authentic assessments into the program? How do you avoid "teaching to the test?") Which is better, an administrative program or a choice program? Should vouchers be based on need? Does money matter in helping kids in high-poverty areas reach high standards?

The author draws a number of conclusions. First, schools are the most effective units of accountability. Second, attempts to adjust for socioeconomic status have been publicly controversial. For example, in MI, the state's Black Caucus opposed any adjustments to the district-level outcome measures that would set lower expectations for African American children. Moreover, if all the multiple goals of a school can't be measured and rewarded equally, then an outcome-based system has the danger of warping the "complex fabric" of a school by prioritizing one goal over another. Consequently, the authors conclude that not much is understood about these systems and states should move slowly in implementing them. When comparing administrative and choice programs, the author points to arguments that conclude that the decision depends on the degree to which the type of school impacts a student's learning. If the effect on student learning of attending a particular school is fairly uniform across students, then centralized systems are better because there are fewer costs borne by those students who remain behind when others leave. If the effect tends to vary, then choice programs will be more effective.

< back to Contents >

Newmann, F.M., King, M.B., and Rigdon, M. (1997, Spring). "Accountability and school performance: Implications from restructuring schools." Harvard Educational Review, 67 (1), 41-74.

Newmann and King address several issues in accountability, which they consider a key strategy of school-based restructuring efforts. In theory, increased school accountability will enhance student performance, as it pressures teachers to improve their instructional strategies and curricula. Since the schools in the study did not share a common metric for student achievement, no causal relationships could be made between approaches to accountability and performance results; instead, the study relies largely on qualitative data to explore the accountability structures within twenty-four restructuring elementary, middle, and high schools, from sixteen states sand twenty-two districts. This sample of schools was drawn from a recent, larger comprehensive study on school restructuring by the Center on Organization and Restructuring of Schools at the University of Wisconsin at Madison.

According to Newmann and King, a complete accountability system should minimally include: 1) explicit standards for student performance, with consequences for success or failure; 2) information, standards, and or consequences required by an external agent and/or the school itself; and 3) an external agent that required assessment beyond mandatory standardized testing as part of the school's accountability system, or required the school to develop its own accountability system.

Each of the twenty-four schools were examined according to these criteria for accountability; if a school had all components of information, standards, and consequences (either externally required, internally generated, or both) it was ranked as "strong". If a school had none of the four criteria, it was "weak". In addition, schools were ranked according to their school organizational capacity, as defined above.

Evidence from twenty schools shows that: one third of schools had strong external accountability systems; organizational capacity varied considerably, and was not related to accountability; schools with a strong external accountability system tended to have low organizational capacity; and schools with strong internal accountability also had stronger organizational capacity.

Newmann and King conclude that schools which try to operate within a context of strong external accountability sometimes face challenges in organizational capacity, such as in those cases where mandatory requirements conflicted with a school's program philosophy or with individual teachers' orientations. In contrast, the authors found a positive relationship between schools that exhibited strong internal accountability and organizational capacity. Meanwhile, the three schools with low external and high internal accountability had developed their own requirements for accountability, and thus had become very collaborative toward a shared purpose. One school that showed improved student outcomes, Humboldt Elementary (also based on the Accelerated Schools model), had as its goal that all children would score at grade level according to national norms by fifth grade. The entire faculty agreed with this goal, and there was considerable "peer pressure" among teachers to adhere to or work towards this performance standard.

Although unable to empirically relate a school's level of accountability and organizational capacity to student learning, the authors' research does provide a rich qualitative set of data from which to draw inferences. Based on such information, the authors offer guidelines for external agents (e.g., district and state agencies) to set accountability while simultaneously improving organizational capacity. Opportunities include setting expectations that individual schools establish their own standards for performance and a responsible reporting system and another is to support staff development in formulating performance goals and ways to meet them. By offering concrete examples of high standards in specific curriculum areas, approaches to assessment, and reliable ways of evaluating student performance, external agents can offer leadership in the definition of high standards without imposing uniform tests in all schools. Such strategies lend themselves to enhancing organizational capacity. Strong internal systems of accountability, as well as sponsorship and support from external agencies, may help schools build their organizational capacities.

< back to Contents >

Odden, A., Heneman, H., Wakelyn, D.J., & Protsik, J. (1996, October). "School-based performance award designs: A case study." Paper presented at the meeting of the American Educational Research Association Annual Conference, Chicago, IL.

This paper represents the collaboration of the Consortium For Policy Research in Education (CPRE) Teacher Compensation Project and both district and union leaders from various school districts and one state. A series of meetings was held to discuss and develop designs for school-based performance awards. CPRE framed the meetings with their own performance award template that posed design questions in six categories: performance targets, awards, funding, best practices, implementation, monitoring, and evaluation. Participants were also briefed on the comprehensive and well-known Kentucky and Charlotte-Mecklenburg award systems.

Kentucky's program, implemented in 1990, uses an assessment that tests students in grades 4,5,7, and 8 in seven core subject areas. Student scores are rated as "novice", "apprentice", "proficient", or "distinguished". The goal is for every student in the state to reach the "proficient level" within a twenty year timeframe; thus, awards are determined by each school's progress toward meeting that goal. Using an index score to rate performance, a school must make a ten percent improvement plus one more point in the difference between its current index score and the 100-point proficient level every two years. For example, if a school's index score is 40, then it must improve its score by 7 points (6 points is 10% of 100 minus 40, plus one additional point). Awards are paid for by the state, and teachers vote on how to spend the award money. Results indicate that 95 percent of Kentucky's schools have made progress in their index scores, the number of students performing at the proficient or distinguished level has doubled, and overall academic achievement has improved 19 percent.

Charlotte-Mecklenburg's program initiated its Benchmark Goals program in 1991 as part of a larger reform effort towards accountability. Each school has its own set of targets, most of which focus on improving test scores on state assessments, but also include attendance, advanced course enrollment, and dropout rates. Targets can be fully or partially met, for a full or partial award. The authors note that the Charlotte-Mecklenburg plan sets different annual targets for African-American students and for all other students, agreed upon after a dialogue with the African-American community. Results for this program indicate that student achievement has improved, the Black/White achievement gap has shortened, SAT test scores have improved, and enrollment in higher level courses by African-American students has tripled, from 9% to 26%.

Reviewing CPRE's performance award template, participants voiced their concerns and shared ideas. Although concerned with finding continual sources of funding, using valid and reliable assessment instruments, and obtaining teacher buy-in, nearly everyone agreed that performance awards would benefit students and enhance public support for schools, thus benefiting teachers as well.

Components of the designs proposed by the participants included the following: concentrating on improving student achievement in five core academic subjects areas (every district also added indicators such as attendance and dropout rates as a small percentage of the performance measure); focusing on the school as a unit, not on individual teachers; use of current state or district testing programs to measure results, awaiting development of more standards-oriented, criterion-referenced achievement measures as soon as possible; calculating performance improvements in terms of value-added to the past performance, i.e., on improvement or changes produced; including scores for all students except the severely disabled; awarding bonuses to all staff in a school, with awards in the $1000-$2000 range; funding awards through a protected escrow account. The authors mention that only two plans specifically included methods of disseminating best practices, and that only recently have the model programs, Kentucky and Charlotte-Mecklenburg, realized the necessity of spreading knowledge of this nature in order to achieve long-term results. All of the participants agreed that a performance award system should be only one component of a larger reform effort that focuses on results.

< back to Contents >

A New Framework for State Accountability Systems. Special Report of the SouthEastern Regional Vision for Education (SERVE). (Revised, September, 1994).

The SouthEastern Regional Vision for Education (SERVE) was commissioned to study barriers to school reform in the southeast, and found state accountability systems (state-level processes designed to inform the public about student performance) to be central in the issue of school reform. Studying six state accountability systems, the report supports the idea of accountability but stresses the necessity for the educational community to be "ready" to use it. Goals, standards, and assessments need to be firmly established before accountability is implemented.

Some of the problems SERVE found with the state's accountability system are: as centralized mandates, there is little local sense of commitment or ownership to externally-derived goals and standards; state agencies are undergoing overhauls from a regulatory to a supportive role, and cannot provide the support and leadership necessary for adequate assistance; generic professional development practices have been inconsistent with the efforts needed for individual schools; consequences are perceived as meaningless in historically low-performing schools. Although all six states are transitioning from old (regulatory) to new (supportive) frameworks and assessment processes, and many are networking with low-performing schools, the authors caution that it is easier to begin an effective accountability system than it is to make mid-course corrections.

The authors believe that state-wide accountability systems should not be implemented until: (1) there is a consensus around goals, standards, and assessment, as well as enough time and funding for training, staff development, and local planning; (2) new standards and assessment practices have the support of practitioners who will implement them; (3) states perform needs assessments to determine the types of resources necessary for each school system; (4) schools build higher capacity by establishing links between education and business leaders, economic development efforts, social service providers, and other organizations in the community; (5) states provide for necessary resources, particularly in regards to teachers' time, and call on local school systems to design their own staff development plans that are congruent with new accountability standards, adhere to "best practices", and are themselves accountable; (6) accessible and responsive support organizations are located that provide tangible services; (7) site-based accountability systems are developed that review effectiveness on a more comprehensive scale than test scores alone; (8) systems of higher education are reminded of their responsibility to contribute to the public good by assisting school districts to build their capacity.

< back to Contents >

Suarez, Tanya M. et al. (1991, June). "The Use of Sanctions with Low Performing School Districts." North Carolina Educational Policy Research Center, Chapel Hill.

This article provides an overview of state takeovers and summarizes the experiences of New Jersey and Kentucky. Recommendations are put forth for the North Carolina State Board of Education as it develops its own policies regarding sanctions for low-performing districts. State takeovers are considered the most severe form of sanction imposed on a low-performing school or school district, and are rarely done in practice. Analogous to financial bankruptcy, persistently low student performance to established standards is considered "academic bankruptcy" if it does not improve within a given amount of time. Versions of school district takeovers have occurred in the 1960s in order to achieve desegregation, and specific components (usually financial) of operation have received takeovers, sometimes when districts themselves request it. As a penalty-based sanction, however, use of state takeovers has only occurred in New Jersey and Kentucky, though legislation for it exists in eleven states. Three states extend takeover policies to individual schools within districts.

Following an analysis of problems, strategies for improvement such as developing and implementing a plan by the district, or providing technical or other assistance to the district, are implemented with more or less local involvement depending on the state. In most state statutes, however, the state accepts these responsibilities. When all attempts have failed, the ultimate sanction is the removal of the district's leadership. States vary as to the fiscal responsibility of the actions to be taken during sanction.

A summary of Kentucky's experience reveals that in 1984 Kentucky implemented an educational deficiency program, requiring the submission of performance reports to the State Department of Education and the local public, and requiring the creation of a Master Educational Improvement Plan. The program outlined nine areas of possible deficiency, including failure to comply with required reports and timelines, and failure to meet performance standards on standardized tests, attendance and drop out rate. Four phases of compliance included: Phase I for districts with no deficiencies; Phase II for districts out of compliance with timelines established in their own improvement plans, with these districts eligible for, but not required to receive state technical assistance and recommendations for improving deficiencies; Phase III was reserved for persistent noncompliance from districts; Phase IV districts were subject to removal of superintendents, local board members, and other district personnel. In January 1989, two Phase II districts were moved to Phase III, because of insufficient improvement in attendance rates. One county was also non-compliant in over 90 criteria. Intervention for these Phase III schools included a curriculum audit, interviews with the local school board school personnel, parents and other related persons, site visits, management assistance, technical assistance, and direct supervision by Department of Education staff on a daily basis. Local school board decisions were subject to approval by the state superintendent. The state provided substantial technical assistance; however, teachers complained that the intervention took place exclusively at the district level and was not felt at the school level. Some changes were appreciated, however, such as a new district policy manual and audit recommendations concerning budgeting.

The second county given Phase III status failed to improve sufficiently in both the attendance and drop-out standards. Serious management and financial problems were also found. The county filed and won a suit against the state, the courting finding that the state modified standards for deficiency that singled out these two counties from other districts with equal deficiencies; that the state did not provide the technical assistance to the districts as specified by its laws; and the legislature had not funded the Educational Improvement Act sufficiently to permit the realization of program goals.

Another court case ended the educational deficiencies program altogether, when 66 districts won a suit against the state educational system, for violating the "state's constitutional mandate for the provision of an 'effective system of common schools'." The result was sweeping reforms of public education in Kentucky, including: new state curriculum, new assessment procedures, prohibition of nepotism, expanded teacher involvement, newer and more equitable funding distribution for districts, and a system of rewards and sanctions.

The authors conclude that considering the hostility, resentment, and ultimate failure of Kentucky's takeover process, there are several lessons to be learned: The criteria for decisions to apply sanctions must be unambiguous and administered equitably across districts, and the criteria used to justify a takeover be decided upon for that purpose alone. They must involve critical indicators that emphasize outcomes rather than process, with the extent of shortfall meeting the test of "reasonableness".

< back to Contents >