View/Download the PDF (Adobe Acrobat Reader Required)

Case Study: A Comparison of the Confidence-Based Learning™ Methodology to Traditional Learning Methodologies on the Patient Safety Knowledge of First-Year Medical Students

By David Mayer, MD
Associate Dean, University of Illinois at Chicago College of Medicine
Co-Executive Director, UIC Institute for Patient Safety Excellence

Abstract
Faculty members at the University of Illinois at Chicago (UIC) College of Medicine set out to determine the effectiveness of knowledge acquisition in patient safety education using the Confidence-Based Learning methodology compared to more traditional learning methodologies. In November 2008, a randomized, double-blinded, cross-over control study was conducted with 128 first-year medical students, implementing a Confidence-Based Learning-based curriculum specially designed and developed by Transparent Learning Inc., partnering with Transparent Health, Colorado Foundation for Medical Care (CFMC), Zipline Performance Group and Knowledge Factor. Results of the study found that the students who were administered an online Confidence-Based Learning module were shown to be 15% to 20% more effective in knowledge acquisition vs. the students who used traditional learning methodologies.

Situation
Advanced learning methodologies, such as Confidence-Based Learning™1, are often seen as alternatives to traditional methodologies in teaching patient safety to first-year medical students. However, no examples of control studies appear in the literature that demonstrate Confidence-Based Learning’s effectiveness on knowledge acquisition in this setting. The University of Illinois at Chicago (UIC) College of Medicine, a leader in patient safety education, was the site of a 2008 study to compare the Confidence-Based Learning methodology to traditional learning methodologies. The Confidence-Based Learning methodology is designed to take any individual to a state of mastery over a shorter time than traditional training methods. It accurately measures the knowledge quality of the learner, isolating confidently held misinformation and doubt, and quickly and effectively remediating knowledge and confidence gaps until the learner has attained mastery. By doing so, the Confidence-Based Learning methodology reduces the risk of mistakes, significantly reduces training time, and improves performance.2

Figure 1: Confidence-Based Learning Behavior Model

The Confidence-Based Learning methodology is based on a unique learning behavior model (Figure 1) that moves the learner toward mastery after determining areas where he or she is misinformed, uninformed, or in doubt. The methodology starts with a formative assessment to measure each learner’s knowledge and confidence.

For each assessment item, the learner can select from one of three confidence levels:

• I AM SURE – A, B or C (learner is confident and shows either mastery or misinformation),
• I AM PARTIALLY SURE – a combination of two answer choices: A or B, B or C, or A or C (learner is doubtful when correct, or misinformed when incorrect), or
• I AM NOT SURE (learner is uninformed).

Immediately upon submitting the assessment, targeted feedback is provided to the learner, who then focuses on those areas of learning where he or she has knowledge and/or confidence gaps. This process is repeated until mastery is achieved.

The Confidence-Based Learning methodology was implemented in this study to establish its effectiveness for first-year medical students related to the learning acquisition of fundamental patient safety knowledge. Patient safety education focuses on the avoidance, prevention, and amelioration of adverse outcomes or injuries caused by the processes of providing patient care. It is fundamental to health care practice and is a common goal of all health science professionals. Because the goal of patient safety education is to help prevent or at least reduce life-threatening medical errors while increasing quality care outcomes, medical students must acquire and master the knowledge, skills and behaviors important to the safe provision of care so they can apply these competencies as health care providers with a high degree of confidence. The Confidence-Based Learning methodology was used in the study to determine whether students could master basic patient safety knowledge more effectively when compared to traditional learning methodologies.

Patient Safety Curriculum
The Confidence-Based Learning methodology used in the study, the two-module Introduction to Patient Safety, was provided by Transparent Learning, Inc. (TLI), based in Denver, Colorado, in partnership with Transparent Health, Colorado Foundation for Medical Care (CFMC). Zipline Performance Group, and Knowledge Factor. Transparent Health and CFMC provided expert consultation relating to the application of Confidence-Based Learning to health care, Zipline wrote and produced the Confidence-Based Learning program, and Knowledge Factor developed and supported the methodology and technology. TLI designs, develops and delivers innovative patient safety and quality improvement educational programs to health care providers and patients. Through its educational programs, TLI aims to improve patient care outcomes by addressing current gaps in continuing health science, graduate, undergraduate and consumer education. TLI’s mission is to expand the knowledge of all health care providers and patients through outcome-based educational programs that reduce medical errors, lower risk and improve patient care outcomes.

Introduction to Patient Safety conveys timely knowledge on three critical Institute of Medicine (IOM) safety/quality reports that are considered the bedrock of the modern patient safety movement: To Err Is Human: Building a Safer Health System, Crossing the Quality Chasm: A New Health System for the 21st Century, and Preventing Medication Errors. Among the learning objectives, the student would learn to:

• Describe different types of medical errors seen in practice.
• Describe the methods outlined in the IOM report To Err Is Human in reducing medical errors.
• Define the IOM’s six aims for health care.
• Describe and discuss the IOM’s 10 rules for redesign.
• Discuss the four main areas of redesigning the health care delivery system.
• Describe methods to reduce medication errors.
• Define the missions and work being done by The Joint Commission.

Study Hypotheses
The UIC College of Medicine study was designed with the following hypotheses:

• First-year medical students using the Confidence-Based Learning methodology score statistically higher on patient safety knowledge-based exams compared to students using traditional learning methodologies on the same exams. This research question is about scores on exams using two approaches in learning.
• The Confidence-Based Learning methodology will identify those first-year medical students who are reluctant to answer "I Don’t Know" and subsequently answer a patient safety knowledge question with some confidence but are wrong (misinformed) about their understanding. This research question is based on identifying guesswork in the learning process and the use of quadrants in attaining mastery.
• The Confidence-Based Learning methodology will identify those students who are honest in their learning and will be able to apply the knowledge better than those using traditional learning methodologies. This research question is based on the speed to competency (learning) using the Confidence-Based Learning methodology vs. traditional methodologies.

Study Process
In early November 2008, a randomized, double-blinded, cross-over control study was conducted with 128 first-year students of the UIC College of Medicine. Three steps were included in the study process: 1) students were asked to read and critique four patient safety articles, 2) after reading all four articles, students were required to take a Confidence-Based Learning module based on two of the four articles, and 3) students were required to take a surprise multiple-choice question (MCQ) test to assess knowledge acquisition on all four articles. Each step was completed before students were required to attend a scheduled two-hour patient safety education workshop on the material.

1. Review of four articles: A week before the workshop, students were required to prepare for the workshop by reading four patient safety review articles: the executive summaries of the three aforementioned IOM reports, as well as a paper describing the mission, patient safety goals and patient safety programs of the Joint Commission.

2. Administration of a Confidence-Based Learning module based on two of the four articles: After reading the articles and prior to the patient safety workshop, nearly all students were administered one of two Confidence-Based Learning modules, each containing 35-40 assessment items. The students were divided into three groups:

• Half of the students were randomly assigned to complete a Confidence-Based Learning module that focused exclusively on two of the four review articles (Group 1).
• Another half of the students were randomly assigned to complete a Confidence-Based Learning module that focused exclusively on the other two review articles (Group 2).
• In addition, 13 students not placed in the two previous groups were not exposed to Confidence-Based Learning prior to the workshop, but only reviewed the four articles. This served as an additional control group (Group 3).
• Students in Groups 1 and 2 had to continue working on their respective Confidence-Based Learning modules until achieving a mastery score of at least 80% correct. While the amount of time required to successfully complete the modules varied, 99% of all students continued working until they achieved at least the minimum mastery level.

3. Administration of an MCQ test on all four articles: Immediately prior to the start of the workshop, students were administered a surprise MCQ test on the four review articles, with five questions derived from the content from each of the four articles. For Groups 1 and 2, 10 questions on the MCQ test related to content learned through a Confidence-Based Learning module, while the other 10 questions related to content learned through reviewing the articles. Students were allowed 20 minutes to finish the test.

Study Findings
Scores on the MCQ test were used to identify and evaluate possible differences in knowledge acquisition between the Confidence-Based Learning methodology and the traditional learning methodology on both a group and individual basis. The results showed a statistically significant difference between groups using the Confidence-Based Learning methodology vs. the traditional learning methodology. The cross-over study design verified that this difference was not related to inherent differences between the two groups. In the statistical analysis (see Appendix), Test 1 refers to the MCQ questions related to Group 1’s Confidence-Based Learning module, while Test 2 refers to the 10 MCQ questions related to Group 2’s Confidence-Based Learning module. The mean (or average) scores for the MCQ questions related to each group’s Confidence-Based Learning module (i.e., Test 1 for Group 1, and Test 2 for Group 2) are significantly higher than the mean scores for MCQ items related to the each group’s traditional learning.

Conclusion
Based on the results of the randomized, double-blinded, cross-over control study with first-year medical students at the UIC College of Medicine, the Confidence-Based Learning methodology was proven to be 15-20% more effective in knowledge acquisition of basic patient safety education over traditional learning methodologies. To determine if there is any difference in knowledge retention using the Confidence-Based Learning methodology over traditional learning methodologies, all students will be administered another patient safety MCQ test several months later. This test will assess critical knowledge that should have been acquired and retained. Scores will be broken out and evaluated to determine possible differences in patient safety knowledge retention between the Confidence-Based Learning methodology and traditional learning methodologies on both a group and individual basis.

References
1 The Confidence-Based Learning System is a patent of Knowledge Factor, Inc. (U.S. Patent #6,921,268)
2 Tim Adams and Brian Webster, “Retention and Confidence: The Impact of Confidence-Based Learning on Knowledge Retention,” Knowledge Factor white paper, ©2007 Knowledge Factor. Available at www.knowledgefactor.com.

Appendix: Statistical Analysis