Part IV
Guidelines for Implementation

This section is designed to guide organizations in using the NHCSSP Standards. Drawing on lessons learned from the pilot site studies, we present guidelines for three types of applications, each of which is part of the larger effort through which students, workers, and employers work toward career development.

  • curriculum development and design;
  • employee recruitment, hiring, and training; and
  • assessment of students and workers.

    Regardless of the specific standards application an organization intends to pursue, asking the following three questions at the outset will help it begin:

  • What are our goals and how can the skill standards help achieve them?

  • What are our current systems (teaching, hiring, assessment) and resources (personnel, funds) relative to our goals?

  • What, if anything, is missing and needs to be created in order to accomplish our goals?

    What follows is intended to illustrate some overall principles for application of the standards, not to prescribe specific courses of action. The first three sections describe general models and initial steps to be taken in the process of translating skill standards into practical tools. The final section of this chapter contains general recommendations, garnered from our pilot study experiences, for putting the standards into practice.

    Curriculum Development and Design

    Job restructuring is a fact of life in today's continually changing health care industry. New health care roles demand a higher level of skill than ever before. Anticipating the increasing breadth of knowledge required by health care workers, the NHCSSP Standards have been broadly drawn and reflect higher order thinking and performance skills. To train future health care workers to meet these standards, educators in all settings must rethink the ways in which they design curricula and courses. While curriculum development and design is of primary interest to the education community, instruction is also integral to the work of trainers, private consultants, and labor organizations.

    A number of curriculum development models exist for health care occupations. Most represent curriculum as the confluence of three major factors: the functions, or technical aspects of care; the health care worker's role, or the interpersonal aspect of care; and the context, or setting in which care is delivered. The curriculum designer must be sure to incorporate these elements into courses, either implicitly or explicitly.

    Since the NHCSSP Standards are neither basic skills checklists nor duty-task lists, but are instead broad statements of what students or workers should know and be able to do to provide quality health care, they provide a template or starting point for the development and design of curriculum. The standards statements can, for example, be developed into learning units to create a course outline, with the points (i.e., sample applications) listed underneath serving as modules to be covered. Users should bear in mind that the lists of points are not all-inclusive but, as with all the components, should be expanded and tailored to suit individual purposes.

    Furthermore, there need not be a one-to-one correspondence between skill standards and learning units. A single standard may contain multiple themes that can become different units. Conversely, one learning unit can incorporate multiple standards, or even different levels of standards (Core plus Cluster). During the implementation study, pilot sites that designed new units based on the NHCSSP Standards found that they were able to cover multiple standards in most of their units. For example, the Stevenson Area Vocational-Technical Education Systems (SAVTES) found that seven of the eight NHCSSP Core Standards could be covered by the course objectives of a Clinical Ethics unit. St. Petersburg Junior College found that requiring the students to complete a nursing care plan on a patient as part of their workplace learning addressed not only the NHCSSP Health Care Core Communication and Teamwork Standards, but also three of the five Therapeutic/Diagnostic Core Standards (Client Interaction, Intrateam Communication, and Monitoring Client Status).

    The development of learning units around the NHCSSP Standards is a complex process. It requires a number of steps, some of which are described in detail below. We present specific examples here that will make the utilization of standards as a curriculum design tool tangible and more relevant for the user. The first five steps are activities for curriculum planners and developers at various levels (e.g., state, district, local), while steps 6 and 7 apply to individual instructors who are using the standards as the basis of a particular course outline and lesson plan. Figure 7 depicts seven steps one can use to develop curriculum aligned to the NHCSSP Standards.


    Figure 7
    Step for Standards-Based Curriculum Design


    1. Establish Project Goals

    The first step in the curriculum development process is establishing goals. Determine what you want to accomplish with a new or revamped curriculum. The following are examples of general goals for curriculum design or redesign that were identified by some of the pilot sites:

  • update curriculum to align more closely with changing industry needs;

  • revamp curriculum to highlight problem solving and thinking skills; and

  • incorporate work-site learning components.

    2. Examine the Standards

    The second step is to examine the NHCSSP Standards, culling out the major ideas or themes that lend themselves to translation into learning units. Identify discrete modules within each, then delineate specific topics for each module. Take, for example, the Health Care Core Standard on Safety Practices:

    Health care workers will understand the existing and potential hazards to clients, coworkers, and self. They will prevent injury or illness through safe work practices and follow health and safety policies and procedures.

    The major themes are "understand existing and potential hazards," "prevent injury or illness," and "follow health and safety policies and procedures." Figure 8 below illustrates how the first theme could be further delineated into specific components.


    Figure 8
    Identifying Learning Units and Modules

    Unit 1: Existing and potential health and safety hazards
    ("Understand existing and potential hazards")
      Module A: Fire and Electrical Safety
      ("Prevent fire and electrical hazards")
        Topic 1: "Fire Prevention"
        Topic 2: "Fire Emergency Plan"
        Topic 3: "Electrical Hazards"
        Topic 4: "Protocols"
        Topic 5: "Preventive Measures"

      Module B: Hazardous Materials
      ("Manage hazardous materials")
        Topic 1: "Materials Identification"
        Topic 2: "Materials Handling"
        Topic 3: "Materials Disposal"


    The two remaining themes may also be expanded into instructional units. "Prevent injury or illness" could form a unit under which the point "Use Universal Precautions to control the spread of infection" could serve as a module entitled Universal Precautions, whose topics would include "Body fluids" and "Protective equipment and clothing." Additional modules under "prevent injury or illness" could be Asepsis and Isolation procedures. Finally, the last theme, "Follow health and safety policies and procedures," could be taken as a unit under which Emergency procedures (from the point "Follow emergency procedures and protocols") would be a module and for which a skills topic might be "Cardio-Pulmonary Resuscitation (CPR)." Thus, themes embodied in the standard statements can correspond directly to major units of learning, while the listed points can offer possible modules from which topics can be developed.

    3. Consult with Industry Partners and Labor Organizations

    To ground the work in actual practice, establish a curriculum design committee composed of representatives of local health care employers and unions. This committee should help predict and facilitate adjustment to current local, regional, and national trends in the industry; identification of health career occupations for which local employers want to hire; and identification of the skills that workers should have. The committee can also provide information regarding the informal culture and nuance of various industry and labor organizations. Finally, it can provide feedback on newly-developed frameworks or curricular materials.

    Similarly, establishing a partnership between educational and health care institutions can yield useful information about the current and future status of the industry. For example:

  • pilot site Elizabeth Street Learning Center partnered with Kaiser-Permanente Bellflower Medical Center to provide workplace learning opportunities for students;

  • SAVTES used a task analyses (DACUM) process conducted by local health care practitioners to identify core competencies in health care that provided independent evidence of the validity of the NHCSSP Standards;

  • teams of teachers in the Pittsburgh Health Careers Initiative performed job audits within the various local delivery settings to see if and to what extent workers performed tasks that corresponded to the NHCSSP Core and Cluster Standards.

    As the health care arena continues to undergo significant changes, such communication and interaction between educators and industry ensures that curricula will be constantly re-evaluated and updated.

    4. Investigate Existing Materials

    Investigate existing data sources, reference materials, and textbooks to help identify the tasks, skills, or competencies that should be included in the intended framework or course. Source materials should include databases established by state or other organizations, task analyses for various occupations, job descriptions from local health care facilities, scopes of practice and competencies from relevant professional associations, as well as current articles in periodicals and journals. Additionally, national, state, and local regulatory requirements that pertain to particular domains must be considered. Such research of existing knowledge will provide a strong foundation on which to build. By taking advantage of existing materials as a basis for curriculum development, one can avoid duplication of efforts and enhance curriculum design.

    5. Identify Learning Outcomes

    Based on information gathered from the previous steps, identify learning outcomes that specify behaviors, performance conditions, and evaluation criteria. Two examples of learning outcomes for specific modules are shown below.

    Module Learning Outcome
    Fire and Electrical Safety "Upon completion of Topic X, participants should be able to locate emergency exit routes and describe evacuation procedures in a nursing home."
    Universal Precautions "Upon completion of Topic Y, participants should be able to explain the importance of handwashing and gloving and describe each procedure."

    One of the pilot sites created and tested a new module using Information Services Cluster Standards. Students learned about client hospital bills by calculating such bills from itemized charges, categorizing new clients according to payer, and solving client problems. The 13 learning outcomes developed for the unit covered four NHCSSP Information Services Standards (Analysis, Abstracting and Coding, Documentation, and Operations) and three Health Care Core Standards (Academic Foundation, Employability Skills, and Systems).

    The pilot sites that undertook curriculum design either examined their existing curricula to determine the extent to which it corresponded to the NHCSSP Standards or actually designed new curricula using the Standards. Sites that compared their curriculum to the Standards found, for the most part, that it either included specific outcomes or objectives that matched the Standards or that the content of the Standards was otherwise integrated within their coursework. For example, Longmont United Hospital found that many of the NHCSSP Standards were being covered in both their CNA and EMT-Basic curricula and that most units covered multiple standards. Several sites created matrices to keep track of the Standards in their various learning units, modules, and specific learning outcomes. An example of such a mapping matrix is show below in Figure 9.


  • Figure 9
    Integration of National Health Care Skill Standards into a Health Career Path Course:
    Excerpt from a Sample Matrix

    6. Create Course Outline

    Taking the breakout of the themes obtained in Step 2, and assembling all the information gathered in steps 3 through 5, structure a course outline. It should indicate the themes or general units that the teacher must cover, in addition to the more specific modules, topics, and lessons. These lessons should be directed at knowledge and skills associated with the standard theme.

    For example, the theme "prevent injury or illness" can be transformed into a unit composed of modules including The nature of micro-organisms, Asepsis, Universal Precautions, and Isolation procedures, which are directly or indirectly represented in the points listed under the Safety Practices Standard. For each module, specify relevant knowledge and skills in a course outline. Skills and procedures that illustrate knowledge and understanding of the above-mentioned modules would include, for example, handwashing techniques, proper glove application and removal, double bagging to remove items from an isolation room without spreading pathogens, and unit-specific sterilization procedures. An expanded section of a possible course outline is presented in Figure 10.



  • Core Standard: SAFETY PRACTICES
    Health care workers will understand the existing and potential hazards to clients, coworkers, and self. They will prevent injury or illness through safe work practices and follow health and safety policies and procedures.

    Figure 10
    Excerpt from a Sample Course Outline


    7. Create Individual Lesson Plans

    Using the course outline created in Step 6, together with the learning outcomes from Step 5, the final step in curriculum design is to create lesson plans that specify the best way to teach the material. After specifying the skills or functions that must be mastered, identify the particular context in which the student will be learning or working. To do this, the following three questions must be answered:

  • How will the worker's role be defined (by traditional job categories; by major tasks)?

  • In what environment will the work be performed (skilled nursing facility, clinic, etc.)?

  • For what group(s) of patients will any procedures be performed (pediatric, geriatric, surgical or nonsurgical patient, etc.)?

    Ascertain what academic content foundation is needed to support the acquisition of the knowledge and skills identified in Step 6. For example, knowledge of the human body systems is a significant part of the academic knowledge that supports the skill (or function) of performing CPR. Students may develop this knowledge in the classroom from a teacher's lecture, through role-playing or simulation, or by participating in a cooperative learning activity, and they may gain it by demonstration or hands-on activity in the workplace. The format for the lesson plan will vary because it must be consistent with local student needs and culture. Part of a possible lesson plan for teaching the Safety Practices Standard is illustrated in Figure 11 below.


    Figure 11
    Excerpt from a Sample Lesson Plan


    The next consideration is developing instructional or assessment activities linked to the lesson plans. For example, consider the use of a skills checklist both as a method of determining student progress and as a final assessment of the desired outcome. Using the Monitoring Client Status Standard from the Therapeutic/Diagnostic Core, consider the point, Measure and report client vital signs or other indicators of health status. A skills checklist for taking a client's blood pressure may explicitly detail the component tasks of the process. A list itemizing the first steps would include:

      Blood Pressure Checklist Pass/Fail
    1. Assemble equipment
    2. Clean stethoscope earpieces and bell/disk with alcohol
    3. Wash hands
    4. Identify patient
    5. Explain procedure
    6. Apply blood pressure cuff on bare arm
    7. Center cuff above elbow over brachial artery
    8. Determine palpatory systolic pressure

    The above skills checklist may be used for skills practice by pairs of students who wish to test one another or as a tool for evaluating a student's demonstration. The same checklist can be used by the instructor to assess student mastery (pass/fail). Other assessment strategies are detailed comprehensively later in this chapter.

    Summary

    The NHCSSP Standards were developed by using task analyses, scopes of practice, guidelines, and other materials from a wide range of allied health occupations; and they have been tested for different purposes at various pilot sites. Translation of the standard statements into curricular units can be accomplished at secondary and postsecondary levels of education to produce general frameworks, course sequences, and/or individual courses. Furthermore, the standards can be used to modify existing curricula or to design completely new units of instruction. Although the education community may be the primary audience for using the NHCSSP Standards as a template for course design, consultants, labor unions, and trainers can adapt the process outlined above for the creation of instructional modules for inservice training.

    Employee Recruitment, Hiring, and Training

    The NHCSSP Standards can be used by health care organizations as a framework for the entire process of human resource management and development, including employee recruitment, hiring, training, and retraining. There are important similarities between the processes of developing curriculum and of developing job training programs based on the NHCSSP Standards, but the variances in setting and target group call for a different approach.

    In applying the Standards to hiring and training, health care organizations may decide to focus more heavily on Cluster Standards than on Core, because the former define specific functions. For instance, one pilot implementation site determined that workplace learning units (such as a diagnostic imaging procedure) tended to be based on the Cluster Standards rather than the Core, with some tasks simultaneously covering more than one standard.

    Figure 12 shows an integrated model for applying the standards to human resource management and development, with the employer drawing from a pool of either applicants or redirected and redeployed workers. Although the standards are used for different purposes at each stage, they provide a singular and unifying focus throughout the process. In this way, duplication of work is avoided and consistency is assured. Although this section is directed primarily toward managers of health care systems and, particularly, human resources personnel, labor organizations will also find it helpful in designing career development programs for their members.


    Figure 12
    Employee Recruitment, Hiring, and Training


    1. Background Analysis

    In this stage, the NHCSSP Standards are used as a reference for developing a set of job descriptions, which, in turn, establish hiring criteria for new positions. Background analysis consists of three steps:

  • Conduct a needs assessment of the client base served by the particular health care facility.

  • Identify the critical functions needed to deliver appropriate care to that clientele.

  • Match the relevant NHCSSP Standards and points to the identified functions.

    As an example of the third step, the Environmental Services Standard on Aseptic Procedures includes the point: Prevent the spread of pathogens by cleaning, disinfecting, or sterilizing surfaces, instruments, and equipment. This statement can be broken down into the following tasks: decontaminates instruments, supplies, and equipment according to established policies and procedures; cleans, assembles, wraps, sterilizes, and stores supplies and instrument trays; and sterilizes instruments, equipment, and supplies as required. These tasks become the foundation of competency-based job descriptions. In creating the descriptions, existing legal and licensure requirements must be kept in mind.

    2. Creating Hiring Criteria

    In this second stage, the employer first refines the job descriptions developed in Stage 1. Recruitment and hiring efforts can be directed at two groups: new applicants applying from outside the institution and current employees applying for different positions. Any important employee specifications unique to the particular health care organization should be identified at this point. Some facilities may want or need to hire multiskilled employees; others may have religious or union affiliations that help their hiring policies.

    Traditionally, most organizations hire on the basis of an applicant's education and/or professional license. These criteria, as well as an applicant's previous experience, can be compared to or mapped against the NHCSSP Standards both for classifying applicants and for creating specific screening tools and hiring criteria. For example, in the hiring of laboratory technicians, the NHCSSP Core Standard Academic Foundation and, more specifically, knowledge of chemistry can be incorporated into a specific hiring criterion.

    Similarly, human resources personnel can cross-reference applicants' previous job experiences as presented in resumes by identifying keywords from the language of the standards. In this way, large numbers of resumes and cover letters can be efficiently scanned for those keywords. If NHCSSP Standards are used to define both job descriptions and screening tools, the process of hiring can be streamlined.

    In Figures 13 through 16, job descriptions and resumes are cross-referenced to NHCSSP Standards. The first job description and resume are for the position of central service worker, requiring skills in both the NHCSSP Core and the Environmental Services Cluster Standards. In this case, the worker gained valuable experience in Environmental Operations while working in a motel, as well as in a previous central supply position. The worker also gained experience in teamwork and communications -- critical Core Standards -- while working at a school. The second job description and resume are for a medical technologist, requiring skills in the NHCSSP Core, the Therapeutic/Diagnostic Core, and the Diagnostic Cluster Standards. This worker's resume shows extensive experience in diagnostic functions, indicating a potential match to the job description.

    Using the standards can benefit both workers and employers. The worker can better highlight and explain his or her skills by knowing the standards that apply to an open position and using that knowledge when preparing a resume or job application. Human resource personnel can more efficiently identify the worker's relevant experience and skills. Thus, matching skills and experience to the NHSCCP Standards should result in a successful job placement.


  • Figure 13
    Job Description

    Figure 14
    Resume

    Figure 15
    Job Description

    Figure 16
    Resume

    3. Employee Orientation and Probationary Period

    During any initial probationary period, the worker's knowledge and task performance are assessed on the job. In that context, the NHCSSP Standards can be used to create skills checklists for supervisors and trainees. The first step in creating a skills checklist is to identify the specific functions and role expected of the worker and the context in which the employee will be working. To show how skill standards can be used in building a skills checklist, Figure 17 illustrates how Information Services standards (Abstracting and Coding, Documentation, and Operations) are incorporated into part of a self-evaluation for a Hospital Unit Clerk.


    Figure 17
    Sample Section of Self-Evaluation Skills Checklist


    4. Employee Inservice Training

    Among the many reasons why health organizations invest in inservice training, three stand out as especially important in today's health care environment:

  • increased emphasis on customer satisfaction,
  • interest in developing multi-skilled workers, and
  • move towards team-structured organization.

    Increasing industry emphasis on customer satisfaction and quality assurance has led to a greater need for training in responding to clients from a service-oriented point-of-view. New hires receive this training during their orientation period, while existing employees are encouraged to enroll in special inservice classes. One pilot study participant, a free-standing rural hospital, used the standards as the basis for its Continuous Quality Improvement (CQI) Program. CQI is a systematic approach to quality control that uses objective methods to improve productivity and, therefore, service by integrating customer feedback into problem-solving. Using the NHCSSP Core Standard Teamwork, the medical center offered all staff a series of educational experiences to increase awareness of client safety, from training sessions for managers to work groups for problem-solving various facets of safety. Through the CQI process, the site not only helped reduce preventable client falls, it also improved staff cohesiveness.

    In addition to changes brought on by a focus on quality assurance, trends toward multiskilling or team-structured organization require that workers take on broader and overlapping roles. Here, inservice training will help the employee acquire additional skills. In either case, employers must first ascertain individual worker training needs and then develop the instructional experiences to meet them. The NHCSSP Standards can be used as a framework for both purposes, thereby maintaining continuity with each of the previous stages in the employee recruitment and hiring process.

    In general, the process described in Curriculum Development and Design applies to inservice training. However, while the development of curriculum and of training programs have some common threads, there are notable differences. Industry training is directed at a more specialized audience and is, generally-speaking, more targeted, aiming to address particular gaps in knowledge and skills. Thus, while the process of creating training modules based on the standards parallels that described in the Curriculum Design section of this document, only selected themes and points will be emphasized in any inservice training course.

    5. Skills Improvement and Update

    The final stage in this whole process occurs after applicants have been hired, passed their probationary and training periods, and been added to the workforce. At this point, the primary responsibility for skills improvement efforts shifts from human resources personnel to the individual employee. Here, the worker decides to master current skills and acquire new ones, both of which can result in promotion and advancement up a career ladder. Structural changes over the course of time -- such as the locus of care moving from acute care, in-patient settings to an outpatient setting -- can also compel workers to acquire additional or higher level skills. Furthermore, current employees will need to undergo retraining when labor or skill abundance and shortage become apparent. These changes impact training needs particularly and should be anticipated if inservice programs are to operate effectively. For instance, as computer technology becomes ever more integral to all health care clusters, for everything from client information and inventory management systems to therapeutic and diagnostic tools, both recent and tenured employees will need to update old skills or acquire new ones through retraining or be redeployed to other work areas. As Figure 12 above shows, both promotion -- when an employee advances through mastery of skills -- and redeployment -- when job or task phase-out mandates learning new skills in order to move to another position -- result in a return to stage 2, where skill needs for advanced and lateral occupations are specified. Furthermore, as roles change, so, too, do the desired skill sets change, leading to the necessity of updating the National Health Care Skill Standards.

    Summary

    Employee recruitment, hiring, and training is an integrated process that occurs in five interdependent stages. Figure 12 shows how they interconnect as -- after hiring and the initial orientation -- workers further develop or upgrade skills and are promoted; other workers are redirected laterally (with retraining as needed); and workers are redeployed from an area of skills abundance where layoffs are possible to one of skills shortage where they can be fully employed. The NHCSSP Standards can be used as a tool in each stage of the process, streamlining it and making it more efficient by maintaining a consistency among the job descriptions, hiring criteria, skills checklists and evaluation forms, and training programs.

    Assessment

    Any system of national skill standards is incomplete without serious attention to the issue of assessment. Assessment is the means of determining whether or not the performance of workers and students meet the established standards. Thus, to effectively use the National Health Care Skill Standards as a basis for worker preparation, hiring, retraining, and evaluation the next challenge is to determine the best ways to assess an individual's proficiency on the skills embodied in the National Health Care Skill Standards. This section is intended to provide such guidance to both industry and education users. Because of how critical assessment is to the industry and education sectors, as well as the range of assessment purposes and methods that are applicable to each, this topic receives a more comprehensive and detailed treatment than do the other standards applications.

    The health professions have a long history of assessment and certification. Traditionally, health occupations certification systems have relied heavily on written examinations that used multiple choice questions. Direct evidence of performance has not typically been a large part of the examination, but rather, part of the eligibility requirements to take the certification examination. For example, to be eligible to take the written examination for certification or licensure, radiology technologist and dental laboratory technician candidates must first successfully complete -- and have confirmed by their supervisor or instructor -- a prescribed number of hours of clinical work.

    Over the years, the health care industry has made sustained efforts to include more direct means of assessment in certification and licensure. Since the 1960s, the health professions have worked on developing performance-based assessments that both emphasize real-world context and require examinees to develop and construct a response rather than rely on multiple choice exams. Performance-based assessment methods include written or computer-based clinical simulations, oral examinations, and the use of "standardized patients" (in which non-examinees are trained to portray patients and the examinee is asked to show how he or she interacts with that "patient"). These newer methods, however, have been used primarily for purposes of medical or clinical certification. Their applicability to entry- or technical-level workers has yet to be fully explored. Moreover, practical and technical limitations (e.g., lack of adequate standardization in administration and scoring, prohibitive costs) have often hampered attempts to incorporate these methods in large-scale (e.g., statewide, national) testing programs.

    In the public education arena, the movement toward performance-based assessment is a newer phenomenon, gaining momentum over the last decade or so as assessment in general has increasingly been viewed as an important element of educational reform. Although significant progress has been made in using performance-based assessments to improve instruction and learning at selected sites, overall efforts have received mixed reviews.

    Based in large part on assessment efforts to date in both the health professions and public education arenas, we discuss issues around development and implementation of assessments that are linked to the National Health Care Skill Standards. The following section identifies the key dimensions and methods to consider in developing a system. Different assessment options are presented, including examples of actual tasks that directly relate to the NHCSSP Standards and were developed in our pilot sites or in other-related efforts. The final section describes general principles, rather than a step-by-step process, for developing and adapting standards-based assessment tasks. This is because assessment development and implementation requires specialized expertise that is beyond the scope of this particular document.

    Critical Dimensions of Assessment

    Traditionally, different modes of assessment have been considered appropriate for the workplace versus the classroom. However, in order to find the most valid and cost efficient means of assessment, both educators and industry representatives in health care are now exploring assessment technologies that have traditionally been used in the other's domains. For instance, educators are now examining how such traditional clinical methods as role playing might be adapted for assessment of student performance. For their part, human resource managers are considering the use of brief, written, problem-solving assessment tasks that have more typically been used in education (see description of written scenarios under Assessment Methods), because they can be standardized and administered at a fraction of the cost of the more time-consuming measures of on-the-job performance. Across sectors, practitioners are examining the potential of portfolios or project-based learning as a means of assessing knowledge and applications.

    To help guide readers in choosing the most appropriate means of assessment for their purposes, this document organizes assessment tools according to four interrelated properties, providing both the pros and cons of each option. What follows is a brief description of each property and how it relates to the others. We then describe various assessment methods available to the health care field (in both formal education and industry settings) and their relationship to these four properties.

    Direct vs. Indirect Assessment: Assessment methods vary in the degree to which they directly measure relevant knowledge and skills. Direct measures stress hands-on performance and call for the assessee to create or construct a response to the assessment item or task. By contrast, indirect methods, such as multiple choice test items, require passive responses (recognition and selection of the correct answer rather than construction of a response). Research has shown that indirect methods often suffer from limited credibility as regards their relevance to actual job performance. Moreover, indirect methods often encourage inappropriate means of preparation and coaching (i.e., teaching to the test). While direct methods are viewed as more valid measures, on the other hand, they have been found to be more susceptible to error in scoring (less reliable than traditional means of measurement), more burdensome, and more costly.

    Degree of Burden vs. Disruption: Assessment methods also vary according to how burdensome they are for the assessment administrator and the assessee, and to how disruptive they are to a program of instruction. For example, multiple-choice items and some written scenario tasks are low-burden because they require minimal materials and can be administered in a short, discrete time period. On the other hand, although a multiple-choice test may be less burdensome to administer, it may be highly disruptive. Generally, it is not well-connected to and also interrupts the training process because preparation for the test itself diverts energy from preparation for targeted skills. Conversely, high-burden methods such as long-term projects that may require significant resource allocation for implementation and for professional development of administrators and scorers can be organized seamlessly around skills instruction and training and are thus less disruptive.

    Costs vs. Benefits: Cost/benefit analysis looks at options not just in terms of expense but also in terms of benefits obtained. In applying such analysis to assessment options, it's apparent that some methods, such as role playing or job simulations, are simply more expensive to administer than standard paper-and-pencil methods. Yet their "value-added" to the preparation of workers may far outweigh such costs because these more direct assessment methods better reflect actual work behavior.

    Cumulative vs. On-Demand: Finally, assessments may be characterized based on the amount of time a candidate has to complete its tasks. In this sense, assessment methods fall into two broad categories: on-demand and cumulative. On-demand assessments occur in restricted time periods, during one class period, for example, and thus are easier to monitor and standardize. Cumulative methods, such as portfolios, require longer time commitments. Typically, the assessee is encouraged to revise and seek review of his or her work before it is complete. Optimally, assessment systems should include both cumulative and on-demand components because, in reality, most occupations include some tasks that must be completed in discrete time periods and others that need to be gradually developed and revised over longer periods of time.

    Assessment Methods

    In deciding which assessment method(s) to include in an overall system, it is important to consider where each falls in relation to the four properties and how each method can be tailored to match the user's particular purpose. Described below are several assessment methods that have been or can be applied in the health care context.

    Multiple-choice assessment: Perhaps the most universally familiar assessment method is the multiple-choice test. Largely because of ease of administration and scoring, this method is one of the most efficient and cost-effective means of judging proficiency with respect to skill standards. This methodology boasts a long history of careful empirical development attesting to its reliability and validity for a number of specific applications. Even so, criticism of multiple-choice assessment has grown stronger in recent years as its limitations have been articulated: its indirect connection to actual performance; its tendency to cover subject matter in a fragmented way; and its emphasis on "lower-level" cognitive skills, such as recognition and recall. Some educators are working to improve such tests while others advocate limiting their use or doing away with them entirely. On the improvement side, those developing multiple-choice examinations for medical licensing and specialty certification now attempt to embed items in more realistic contexts, making tests less oriented toward recall of isolated facts. An example of newer multiple-choice items are those that represent simulations of decision-making in client care. Examinees are given detailed description of a clinical situation (including history and physical and laboratory findings) and then asked to indicate the diagnosis, prognosis, or next steps in care.

    On-demand performance tasks: These are assessment activities that require examinees to respond "on the spot," in a limited period of time. Examples of such tasks are direct observations of work on the job. For example, a health care worker might be observed and evaluated on a particular procedure such as bandaging a wound, drawing blood, or running a laboratory test. Performance is rated by one or more trained observers -- supervisors, instructors, or trained raters -- typically using rating forms or checklists. Such samples of performance are usually focused on a discrete set of skills and do not represent the full range of workplace behavior.

    Simulations of work performances are another variant of on-demand performance tasks. Simulations involve a well-defined problem and circumstances for which the examinee must take action. When used for assessment in the health care field, simulations often revolve around hypothetical client encounters. In patient simulation or role playing, the examinee takes the part of the health care worker and a trained participant takes the part of a patient. The role play assessment notes the examinee's professional judgment in decision-making and interacting with the "patient." Role playing has been used at different levels of health care training ranging from candidates for medical licensing for whom standardized patient methodology is a form of role playing to secondary education health careers students who participate in role plays at national and regional competitions.

    Instead of using face-to-face role playing, simulation of professional encounters may utilize a computer-based or written stimulus. For example, at the medical licensing level, patient management problems (or clinical simulations) begin with an opening scenario that provides a brief description of a clinical situation, then requires the examinee to proceed through a series of "scenes" in which additional information is gathered and patient management activities are initiated. The examinee must select the actions to be taken. More generic written scenarios, which can be used with the full range of health care workers and students, present a complex, realistic problem, that is succinctly written, along with a prompt (a question or directive) to which examinees must respond in writing. Written scenarios typically take 30 to 90 minutes to administer. Figure 18 offers an example of a written scenario intended to assess proficiency with respect to specific NHCSSP Core Standards, including Legal Responsibilities, Ethics, Safety Practices, and Communication.


  • Figure 18
    Written Scenario

    Another type of on-demand task is the oral examination, a methodology that has been used for several hundred years to assess clinical skills in both academic and workplace arenas. Within industry, such examinations are now typically used as part of the hiring process. They are administered by human resources or supervisory-level personnel in a set amount of time (30 minutes to two hours), typically with suggested lines of questions and scoring criteria to be used by the examiners. In some formats, examinees are given pre-determined, hypothetical on-the-job cases and situations and then asked how they would handle such situations. Oral examinations attempt to access the same content knowledge and problem-solving abilities as the written scenario, but some examinees might prefer one method over the other. Those with less developed reading and writing skills, for example, might perform better in an oral examination, while those who are shy might prefer a written format.

    One final example of an on-demand task that is particularly useful for assessing performance of administrative and medical assisting health workers is the in-basket exercise. A collection of papers that includes staff schedules, memoranda, letters, purchase orders, and other administrative records is presented to the examinee, who is instructed to demonstrate administrative and supervisory skills by organizing, prioritizing, and delegating work.

    Compared to multiple-choice or objective assessment methods, on-demand, performance-based assessments -- whether actual or simulated -- are more direct and realistic measures of job skills. Among the various on-demand performance-based methods, direct observation of work is, by definition, the most realistic; however, the complexity and lack of standardization in real-life situations often leads to inconsistent ratings among raters or scorers. These problems are only partially diminished by rater training and use of appropriate rating instruments. Simulations offer the advantages of more cost-effective standardization and administration and, therefore, more reliable scoring. However, such advantages come at the cost of having somewhat contrived, rather than natural conditions.

    Performance review (performance assessment): Like workers in other industries, health care workers are subject to periodic reviews, typically by their immediate supervisors. Supervisor assessments are based on explicit criteria of which both the supervisor and employee are aware, such as those inherent in the health care skill standards. The employee should have input into the evaluation process and may be asked to do a self-evaluation. Figure 19 shows an example of a section of a supervisor's assessment for the entry-level position of Physical Therapy Aide. The skills specified here relate to the Core Standards of Communication, Employability Skills, Teamwork, and the Therapeutic/Diagnostic Core Standards on Client Interaction and Intrateam Communication. (Also see Curriculum Development and Design and Employee Recruitment, Hiring, and Training on how the NHCSSP standards can be used to create checklists and evaluations.)


    Figure 19
    Excerpt from a Skills Performance Evaluation


    Peer review: Whereas supervisor evaluations generally involve a one-to-one relationship between individuals (supervisor and supervisee), a peer review is a collective professional judgment of an individual or group performance. Peer reviews can take many forms, but they typically involve periodic meetings during which peers review, discuss, and critique each others work (e.g., cases, care plans, progress notes, portfolios). This review method is becoming increasing popular as shared governance models place power and responsibility in the hands of practitioners. Peer review is viewed as particularly useful because it can reinforce group philosophy and cohesiveness, promote professional growth, and provide a forum for recognition. The effectiveness of peer review is still somewhat controversial in part because team work is still in its infancy in many health services settings. Without adequate structural support and a strong history of working and reviewing performances as a team, peer review makes people uncomfortable. Critics claim that judgments by peer reviewers tend to be unreliable and note, as well, that people working in groups often feel pressure to be amiable and less critical. Moreover, there is evidence that practitioners often feel threatened by peer reviews because they are less accustomed to being evaluated by colleagues instead of a supervisor. Nevertheless, the potential benefits of peer review make it a promising method for further investigation.

    Projects: Assessment projects represent major products or services developed by an individual to showcase his or her proficiency on important job- or career-related skills. Such projects are cumulative assessment tasks that take several weeks or months to complete. Projects have been particularly useful in secondary and post-secondary education because they can be embedded in classroom activities and used to reinforce good instruction on important standards and skills. Projects designed to show proficiency on health care standards might include conducting case studies on selected patients (therapeutic cluster), creating inventory checklists of supplies and equipment for designated facility units (environmental services cluster), or developing public relations materials and brochures for specific health services (health care core). One of the NHCSSP pilot standards implementation sites is having students develop such projects as part of classroom assessment activities.

    Portfolios: Another cumulative assessment method is a portfolio, which is a vehicle for organizing and presenting a collection of work and evidence that demonstrates an individual's skills and abilities. It is developed and shaped over time, sometimes months, sometimes even years. The portfolio is an increasingly popular assessment method in certain professions (health care, architecture, art) as well as among students. In states such as North Carolina, California, and Michigan, portfolios are being used to assist students in school-to-work transition. For example, the Career-Technical Assessment Program (C-TAP) in California is a standards-referenced, portfolio-based assessment system designed to evaluate high school students' preparedness for entry-level jobs and post-secondary training based on their proficiency with respect to state-developed skill standards. The system requires specific pieces to be included in the portfolio, among them: a letter of introduction, a resume, an employment or college application, a letter of recommendation, work samples (demonstrating hand-on mastery of key standards), a writing sample, and a completed supervisor's evaluation form (rating the student's performance in a paid or unpaid work experience). An example of a work sample appropriate for environmental services health care workers is presented in Figure 20. As shown, this work sample demonstrates a student's proficiency on two NHCSSP Environmental Services Cluster Standards: Environmental Operations and Resource Management.


    Figure 20
    Work Sample


    Guidelines for Developing a Comprehensive Assessment System

    A comprehensive assessment system must be flexible, allowing for a variety of tasks and components to meet the goals and specific conditions defined by the user. Goals could range from a relatively informal reporting of a candidate's skills up through formal certification.

    As described above, a variety of options are available for assessing proficiency on skill standards. Many have been in use for a long time, while others incorporate newer assessment technologies. The following guidelines will help readers design an effective and comprehensive assessment system. The guidelines are based on a review of the research and lessons learned from NHCSSP pilot standards implementation sites, as well as project staff experience with related assessment projects.

    Strong links to standards: Assessments must be clearly linked and referenced to standards. A comprehensive assessment system must address all applicable standards within a field, and, if possible, each assessment task should be designed to measure one or more standards. However, it may not be necessary to assess any one individual on all standards, but rather on a sampling of standards. For example, certification decisions for individuals are often based on demonstrated competency across a sample of standards and assessment tasks. Because there are practical limits to the number of assessment tasks an individual candidate can complete, it is important to ensure that a purposefully selective sample of standards is adequately addressed within these tasks.

    Broad inclusion and consensus: As with standards development, assessment development must be a broadly inclusive process, with assessment tasks developed collaboratively by representatives from industry, labor, and education. This will help ensure feasibility in educational and training settings, as well as fidelity to the requirements of the workplace. Many assessment development efforts have underestimated the time and energy needed to reach consensus about what workers and students must learn and demonstrate. Shortcuts in the process opens the assessment system to criticisms of secrecy, irrelevance, and failure to reflect constituent (e.g., industry, labor, education community) needs.

    Technical quality: Whether employing traditional or performance-based assessment systems, technical quality must be a primary concern. Specifically, there must be adequate demonstration of the system's validity and reliability. Validity concerns what the assessment measures and how well it does so. Specifically, it refers to the appropriateness, meaningfulness, and usefulness of the specific inferences made from assessment scores. Reliability refers to the consistency of scores. It indicates the degree to which scores are free from errors of measurement.

    Much criticism of the new methods has centered around their inability to meet traditional criteria for reliability and validity used in multiple-choice assessment systems. In order to use performance-based assessments for accountability purposes, their validity and reliability must be further established. Low reliability, in particular, has been identified as a technical flaw in a number of the new performance-based assessment systems. However, recent studies show progress in this regard. For example, there is evidence that when raters are trained using well-defined scoring rubrics, agreement on ratings by two or more raters on the same assessment tasks can be kept sufficiently high for on-demand performance-based assessments.

    Moreover, there is emerging evidence that performance-based assessments lead to better instruction and training. Such positive "consequences" of assessment use are now being incorporated into new conceptualizations of validity. In short, new language and evaluation criteria are beginning to be researched in order to judge the new methodologies more fairly.

    Research and development must continue on the technical (and practical) aspects of performance-based assessment methods. At the same time, the presumed reliability and validity of traditional approaches must be questioned. (As mentioned previously, limitations of multiple-choice assessment were a major impetus for the emerging popularity of performance-based assessment.)

    Equity: Equity concerns must be addressed throughout the assessment development process. Although some advocates of performance-based assessment made early claims that the newer methods were more equitable, some recent evidence shows that group differences can be exacerbated by new methodologies. Differences in the performance of males versus females or among ethnic and racial groups must be reviewed and analyzed to ensure they are not the result of biased assessment methods. The needs of special populations must also be considered; specific accommodations are often dictated by law or accepted practice.

    Professional development: Professional development or training of assessment developers, administrators, and raters is key to successful implementation of assessments based on any set of standards, including the NHCSSP Standards. Clearly, potential users are at different levels of understanding and readiness for effectively using assessment tools and data. Thus, multiple professional development strategies are necessary to accommodate the needs of organizations, unions, schools, and professional associations. Strategies must include but not be limited to written manuals. Tailored, face-to-face training is an essential component of any comprehensive professional development strategy.

    Variety and flexibility of methods: Optimally, an assessment system should include a variety of assessment components and tasks. Triangulation -- or the accumulation of multiple forms of evidence to reach a final assessment score -- allows for assessment of skills through multiple and complementary means. This increases the reliability of the score and any decisions based on it. For several reasons, assessment systems should include a combination of performance-based tasks, as well as traditional paper-and-pencil (e.g., multiple-choice) items. First, most jobs involve a combination of activities, some that can be completed over time and others that have more immediate deadlines. If they are to be valid and acceptable to the industry, labor, and education communities, assessments must reveal how the worker or student manages in both situations. Second, inclusion of multiple-choice tests in particular can help ensure broader, efficient coverage of standards; this may not occur if only performance-based assessment methods are used. This is because the time- and resource-consuming nature of performance-bound assessment tasks limits the number that can be administered and, hence, limits the number of standards that can be tested. Most important, including various types of assessment components allows examinees to demonstrate competency in different ways, thus revealing an individual's particular strengths.

    Summary

    A major lesson emerging from current assessment efforts is that it takes substantial time and resources to develop and implement systems that include both performance-based and traditional multiple-choice assessment components. Successful assessment implementation in schools, health care facilities, or training facilities requires the intensive capacity development of users to administer and integrate these assessment tasks into their current practices. Moreover, it requires careful nurturing of awareness, involvement, and support from the broader community.

    The complexity of assessment development and implementation -- particularly in light of the emerging technologies -- cannot be overstated. As such, it must be undertaken as a joint effort by assessment specialists and health care constituents. Finally, whether assessment is used for purposes of worker preparation, hiring, retraining, evaluation, or formal certification, success hinges on the ability of industry, educators, and labor associations to build and work in concert.

    General Recommendations from NHCSSP Pilot Sites

    Regardless of the application -- curriculum development and design; employee recruitment, hiring, and training; or assessment -- common themes emerged in the pilot implementation of the National Health Care Skill Standards. Among them, for example, was the need for broad participation in the planning process, both across the organization and vertically, from senior management to line workers. Also, formal mechanisms to facilitate communication among all parties were found to be essential. One site that had no such mechanism found that two departments had made entirely different plans for the target group. At multiple-organization sites, the activities of several groups especially need coordination; such pilot sites found that a single designated agent was needed to shepherd each of the various organizations through the process and synthesize their efforts.

    The following is a list of "general recommendations" made by the pilot sites during the study:

    1. Involve top management in the planning process from the very beginning, ideally working with an individual who can make decisions and motivate others.

    2. Create a planning template for the implementation with timelines and a clear allocation of responsibilities. If possible, designate a single "point person" who has full knowledge of all aspects of the implementation.

    3. Recognize that unexpected developments, such as changes either in the organization (e.g., staffing) or the external environment (e.g., funding source) may impinge on the implementation timeline. Plan for the unexpected, having alternatives from which to choose.

    4. Involve representatives from all departments that will be affected by including them on working committees that meet regularly. Committees should be composed of management, appropriate union representatives, and front line individuals who are responsible for actually implementing the changes.

    5. Because Core Standards represent the fundamental, prerequisite knowledge and skills needed for employment in health care, they must always be the starting point for applications, to be augmented by the more specialized Cluster Standards.

    6. Instructional modules based on occupation-specific, narrow-range standards can limit later mobility of workers who may want to transfer to other areas of health care or to other industries. Modules that are broad in focus and specifically incorporate employability standards may be more useful in teaching the knowledge, skills, and abilities that will remain current and portable into the future.

    7. In curriculum design, create mechanisms for coordinating instructional units with workplace learning by having both the teacher and the worksite mentor review the NHCSSP Standards, map them to the course outline, and meet regularly with each other to compare notes.

    The NHCSSP pilot study represents a small beginning in exploring the myriad possibilities for standards implementation. Project participants at the various sites worked hard during the study window to establish and test real-life applications of the National Health Care Skill Standards. Many sites will actually realize the fruits of their efforts long after this study has concluded. We hope that the process of standards implementation will continue -- in ways we have described as well as new ones -- as localities adapt and tailor the standards to their own purposes.


    [WestEd Home Page | Top of This Page | Contact NHCSSP ]