Everything you Wanted to Know about the CPI: Now you can ask
BY
Sherry Clark, ACCE at Creighton University
Mary Knabb, ACCE at MGH Institute of Health Professions
Ann Jampel, CI at MGH
Janice Meckstroth, ACCE at Bay State PTA program
Debbie Ingram, ACCE at U of TN at Chattanooga
Troy Bourgeois, Past President of APTA Student Assembly
History of the Development of the CPI
Prior to the late 70's and early 80's each Physical Therapist and Physical Therapist Assistant Program had its own Clinical Performance/Evaluation Form.
Some Clinical Education Sites developed their OWN evaluation instrument.
80's & 90's Clinical Education Consortiums began to develop and some "regional forms" were developed.
November, 1993, APTA appointed a 10-member Task Force to "develop consistent clinical education evaluation instruments to measure student performance outcomes in PT and PTA clinical education."
Task Force Members
Physical Therapy members: Sherry Clark,MS, Susie Duesinger, PhD, PT, Barbara Gresham, MS, PT, Pamela Gramet, PhD, PT, Bella J. May EdD, PT, FAPTA, Kathryn Roach, PhD, PT, Babette Sanders, MS, PT
Non PT Members: Rebecca Lewthwaite, PhD, Paul Hagler, PhD
Michael J. Strube, PhD (Consultant)
Design Characteristics
Must measure a performance of multiple skills and behaviors of the PT and PTA on multiple levels (Novice to Entry-Level)
Content and protocol of the instrument must be responsive to the clinical AND academic communities.
Psychometrically sound (valid and reliable)
Process
Task Force met in 1994.
Review of numerous instruments both within and outside the profession (nursing, OT, Speech, medicine) and international instruments (Canadian and Australian).
50 Person Sounding Board was formed to respond to the initial draft of the Task Force.
The first draft was written based on: A Normative Model for Physical Therapist Education, the Guide to PT Practice, Part One, the Evaluative Criteria for PT and PTA Education Programs and reference to the commonly used evaluation tools for clinical performance at that time.
Format
Designed to use Professional Judgment in a complex clinical environment
Performance Criteria were broad with specific Sample Behaviors given to guide the evaluation of the criteria. The Sample Behaviors were just that, a sample, and not all inclusive.
Performance Dimensions were to be used to describe performance of behaviors of each criteria.
Quality
Supervision/Guidance
Consistency of Performance
Complexity of tasks/environment
Efficiency of Performance
VAS
The VAS was chosen as the means of recording the level of observed behavior for each item in the CPI.
Horizontal line of 100 mm represents a continuum of performance from Novice to Entry-Level (Draft 1 had Expert as end point).
Continuous scales can recognize degree of change better than categorical scales. (Clinicians often used the +/- in other scales)
No numbers on the scale made it more likely that meaning would not be attached to the numbers. Use of the VAS scale encourages rating based on the end anchors and not numbers.
Entry-level is defined in each clinical setting and the rater uses professional judgment to determine the degree to which a student meets that criteria.
Instructions for use
Specific instructions were written for the use of the form and that was included in the first draft.
Task Force believed that a therapist should be able to READ the instructions and be able to use the form.
Sounding Board
The sounding board of 50 people were sent the first draft including instructions for use and changes were made in the instrument based on this feedback for the development of Draft 2.
Draft Two
Pilot study of Draft 2 between October 1995 and June of 1996.
Concurrently, regional forums were being conducted by the task force and structured feedback was solicited.
Concurrently, copies of Draft 2 were sent out to academic programs and clinical educators with a feedback form included.
Draft 3
Data from Pilot Study
Feedback from forums
Feedback from multiple communities of interest
Consultation of Psychometrician
Development of Draft 3, the Field Study version.
Draft 4
Current version of the CPI
Based on results of Field Study of Draft 3
Feedback from communities of interest
Consultation of Psychometrician
Changes in terminology of current APTA documents
Dissemination
The Physical Therapist Student version of the CPI was approved by the APTA BOD in November 1997.
The Physical Therapist Assistant Student version of the CPI was approved by the APTA BOD in March 1998.
Currently 95% of PT and PTA education programs have purchased the right to use the CPI
The Development and Testing of APTA Clinical Performance Instruments. Anticipated Publication in April 2002, Physical Therapy.
Roach K, Gandy J, Deusinger SS, Clark S, Gramet P, Gresham B, Hagler P, Lewthwaite R, May BJ, Rainey Y, Sanders B, Strube MJ.
Four major areas of focus:
Clinical faculty training and use.
Interpretation and grading of the CPI.
CPI use in program and curriculum evaluation.
CPI as a tool to facilitate student development and self-evaluation.
CPI Interpretation and Grading
Many variations exist between academic programs.
Performance expectations for various clinical levels
CPI interpretation
"Weighting" of criteria
Grading processes.
CPI Interpretation and Grading
The "Novice to Entry level" VAS moves the evaluation of student performance beyond considering primarily the amount of supervision needed.
Requires a change in the academic "mindset".
Performance standards must be reestablished.
Expected rate of progression to entry level must be considered.
CPI Interpretation and Grading
The specifics of performance issues are not always immediately obvious.
Performance Criteria such as "Treatment/Intervention" (PT version) or "Patient Interventions" (PTA version), cover many component skills and behaviors.
Sample behaviors not all-inclusive
Performance dimensions not broken out
CI written comments:
should indicate the specific performance issues identified.
should be consistent with the VAS.
Wide variation seen in the quality and quantity of CI written comments.
CPI Interpretation and Grading
Tendency for CI "inflation" of student performance continues.
Interpreting the VAS line and comments may necessitate a call to the CI or CCCE.
CPI Interpretation and Grading
Should special significance be given to "red flag" items 1-5?
Should other performance criteria be "weighted" or considered as equal?
What to do with criterion marked as "not observed"?
How to structure grading matrices?
Much can be gained by academic programs comparing approaches.
Program and Curriculum Evaluation
Academic programs need data regarding student performance on clinicals to evaluate and modify curricula.
Program and Curriculum Evaluation
Data gathered from VAS scores may show linkage to the broader categories of the Performance Criteria.
However, the CPI does not provide specific information on the performance dimensions or the knowledge, skills and behaviors involved.
Academic faculty must look for other ways to capture the more specific information needed for thorough curriculum evaluation.
CI comments must also be recorded and evaluated.
Triangulation of data continues to be important.
Facilitating Student Development - The CPI as A Roadmap
Students need a tool to assist them with self-assessment, goal setting, and self-directed learning.
Student memories of verbal feedback and discussions with CI/CCCE can be short and unreliable.
Facilitating Student Development
"Entry level" performance is a multi-dimensional construct and may not be readily understood by students.
Students must be trained in the use of the CPI in order to make use of it as a tool for ongoing development and self-assessment.
