Home About us Contact
|
Home About us Contact | ||
|
|
||
Summative Assessment (summative + assessment)
Selected AbstractsSummative Assessment in Medicine: The Promise of Simulation for High-stakes EvaluationACADEMIC EMERGENCY MEDICINE, Issue 11 2008John R. Boulet PhD Abstract Throughout their careers, physicians are exposed to a wide array of assessments, including those aimed at evaluating knowledge, clinical skills, and clinical decision-making. While many of these assessments are used as part of formative evaluation activities, others are employed to establish competence and, as a byproduct, to promote patient safety. In the past 10 years, simulations have been successfully incorporated in a number of high-stakes physician certification and licensure exams. In developing these simulation-based assessments, testing organizations were able to promote novel test administration protocols, build enhanced assessment rubrics, advance sophisticated scoring and equating algorithms, and promote innovative standard-setting methods. Moreover, numerous studies have been conducted to identify potential threats to the validity of test score interpretations. As simulation technology expands and new simulators are invented, this groundbreaking work can serve as a basis for organizations to build or expand their summative assessment activities. Although there will continue to be logistical and psychometric problems, many of which will be specialty- or simulator-specific, past experience with performance-based assessments suggests that most challenges can be addressed through focused research. Simulation, whether it involves standardized patients (SPs), computerized case management scenarios, part-task trainers, electromechanical mannequins, or a combination of these methods, holds great promise for high-stakes assessment. [source] Reliability of the long caseMEDICAL EDUCATION, Issue 9 2008Tim J Wilkinson Objectives, The use of long cases for summative assessment of clinical competence is limited by concerns about unreliability. This study aims to explore the reliability of long cases and how reliability is affected by supplementation with short cases. Methods, We performed a statistical analysis of examinations held by the Royal Australasian College of Physicians in 2005 and 2006 to determine overall reliability and sources of variance in reliability according to candidate ability, case difficulty and inter-examiner differences. Results, Scores for 546 long cases in 2005 and 773 long cases in 2006 were analysed. In 2006, 38% of the total variation in long case data was explained by variation in candidate ability, with other significant contributors to variance being candidate × case and candidate × examiner interactions. Similar figures were found for the 2005 examinations. A short case is less reliable than a long case, but when examiner time is taken into account, three short cases are as reliable as one long case. Any combination of short and long cases would require 4,5 hours of testing time in order to achieve dependability > 0.7. Conclusions, Long cases can be optimised for reliability but time limits their use as the sole tool in a high-stakes examination. Further examiner training, better case selection, or greater use of short cases would have minimal impact on reliability. Reliability can be improved by either increasing examination time or including additional methods of summative assessment, such as might be provided by workplace assessment. [source] Effects of conventional and problem-based learning on clinical and general competencies and career developmentMEDICAL EDUCATION, Issue 3 2008Janke Cohen-Schotanus Objective, To test hypotheses regarding the longitudinal effects of problem-based learning (PBL) and conventional learning relating to students' appreciation of the curriculum, self-assessment of general competencies, summative assessment of clinical competence and indicators of career development. Methods, The study group included 2 complete cohorts of graduates who were admitted to the medical curriculum in 1992 (conventional curriculum, n = 175) and 1993 (PBL curriculum, n = 169) at the Faculty of Medicine, University of Groningen, the Netherlands. Data were obtained from student records, graduates' self-ratings and a literature search. Gender and secondary school grade point average (GPA) scores were included as moderator variables. Data were analysed by a stepwise multiple and logistic regression analysis. Results, Graduates of the PBL curriculum scored higher on self-rated competencies. Contrary to expectations, graduates of the PBL curriculum did not show more appreciation of their curriculum than graduates of the conventional curriculum and no differences were found on clinical competence. Graduates of the conventional curriculum needed less time to find a postgraduate training place. No differences were found for scientific activities such as reading scientific articles and publishing in peer- reviewed journals. Women performed better on clinical competence than did men. Grade point average did not affect any of the variables. Conclusions, The results suggest that PBL affects self-rated competencies. These outcomes confirm earlier findings. However, clinical competence measures did not support this finding. [source] The effectiveness and reliability of peer-marking in first-year medical studentsMEDICAL EDUCATION, Issue 10 2006Rachel English Background, Peer-marking has been suggested as a method to enhance self-directed learning and reflection, although whether this improves performance is unclear. This study evaluated the impact of peer-marking on examination performance and investigated its reliability and acceptability to students. Methods, First-year medical students were randomised to peer-marking using a model answer or no intervention (control arm). Student scores were compared with tutor-marked scores. Two months later, students completed a summative assessment and performance was compared between students randomised to peer-marking and the control arm. A focus group was held with students in the intervention arm to capture their experiences and attitudes. Results, A total of 289 of 568 students consented to participate and 147 were randomised to peer-marking (142 controls). Students randomised to peer-marking achieved marginally higher examination marks (1.5% difference, 95% CI ,0.8% to 3.9%, P = 0.19) than controls (adjusting for year and in-course assessment), although this may have been due to chance. Students were harsher markers than the tutors. Focus group analysis suggested that students valued peer-marking, although concerns about passing judgement on a colleague's work were expressed. Conclusions, Peer-marking did not have a substantial effect on examination performance, although a modest effect cannot be excluded. Students gained insight into examination technique but may not have gained deeper knowledge. Given its potential positive educational value, further work is required to understand how peer-marking can be used more effectively to enhance the learning experience. [source] Teaching consultation skills: a survey of general practice trainersMEDICAL EDUCATION, Issue 3 2001Alison Evans Background Consultation skills are vitally important in general practice (GP), and now form part of the summative assessment of GP registrars in the UK. GP trainers need to be skilled in teaching consultation skills, and also need the time and resources to ensure that their registrars are competent in consultation skills. Aims To describe the teaching methods used by GP trainers in one deanery, the frequency of teaching of consultation skills, the problems encountered and the training that GP trainers have themselves received both in consultation skills and how to teach them. Method Postal questionnaire survey of all the 164 trainers in the Yorkshire Deanery. Results Replies were received from 129 trainers (response rate 79%) of which 123 could be analysed. Of these trainers, 45 (37%) trainers taught consultation skills fewer than five times a year, 45 (37%) five to 10 times, and 14 (11%) more than 10 times a year. A total of 24 trainers reported problems with teaching consultation skills, most commonly lack of time, technical difficulties, and unreceptive registrars, and 97 (79%) trainers had had some postgraduate training in consultation skills with 112 (91%) reporting some form of teacher training. Conclusion There is considerable variation in the reported frequency of teaching consultation skills, the models used, and the preparation of trainers for teaching, despite a systematic approach to teacher training in the Yorkshire Deanery. [source] Moving Toward a Comprehensive Assessment System: A Framework for Considering Interim AssessmentsEDUCATIONAL MEASUREMENT: ISSUES AND PRACTICE, Issue 3 2009Marianne Perie Local assessment systems are being marketed as formative, benchmark, predictive, and a host of other terms. Many so-called formative assessments are not at all similar to the types of assessments and strategies studied by,Black and Wiliam (1998),but instead are interim assessments. In this article, we clarify the definition and uses of interim assessments and argue that they can be an important piece of a comprehensive assessment system that includes formative, interim, and summative assessments. Interim assessments are given on a larger scale than formative assessments, have less flexibility, and are aggregated to the school or district level to help inform policy. Interim assessments are driven by their purpose, which fall into the categories of instructional, evaluative, or predictive. Our intent is to provide a specific definition for these "interim assessments" and to develop a framework that district and state leaders can use to evaluate these systems for purchase or development. The discussion lays out some concerns with the current state of these assessments as well as hopes for future directions and suggestions for further research. [source] Early identification of ,at-risk' students by the parents of paediatric patientsMEDICAL EDUCATION, Issue 9 2005Maree O'Keefe Introduction, Assessment of medical student clinical skills is best carried out using multiple assessment methods. A programme was developed to obtain parent evaluations of medical student paediatric interview skills for feedback and to identify students at risk of poor performance in summative assessments. Method, A total of 130 parent evaluations were obtained for 67 students (parent participation 72%, student participation 58%). Parents completed a 13-item questionnaire [Interpersonal Skills Rating Scale (IPS) maximum score 91, higher scores = higher student skill level]. Students received their individual parent scores and de-identified class mean scores as feedback, and participants were surveyed regarding the programme. Parent evaluation scores were compared with student performance in formative and summative faculty assessments of clinical interview skills. Results, Parents supported the programme and participating students valued parent feedback. Students with a parent score that was less than 1 standard deviation (SD) below the class mean (low IPS score students) obtained lower faculty summative assessment scores than did other students (mean ± SD, 59% ± 5 versus 64% ± 7; P < 0.05). Obtaining 1 low IPS score was associated with a subsequent faculty summative assessment score below the class mean (sensitivity 0.38, specificity 0.88). Parent evaluations combined with faculty formative assessments identified 50% of students who subsequently performed below the class mean in summative assessments. Conclusions, Parent evaluations provided useful feedback to students and identified 1 group of students at increased risk of weaker performance in summative assessments. They could be combined with other methods of formative assessment to enhance screening procedures for clinically weak students. [source] Prior academic background and student performance in assessment in a graduate entry programmeMEDICAL EDUCATION, Issue 11 2004P L Craig Objectives, This study aims to identify whether non-science graduates perform as well as science graduates in Basic and Clinical Sciences (B & CS) assessments during Years 1,3 of a four-year graduate-entry programme at the University of Sydney (the ,USydMP'). Methods, Students were grouped into five categories: Health Professions (HP), Biomedical Sciences (BMS), Other Biology (BIOL), Physical Sciences (PHYS) or Non-Science (NONS). We examined the performance rank of students in each of the five groups for single best answer (SBA) and modified essay (MEQ) assessments separately, and also calculated the relative risk of failure in the summative assessments in Years 2 and 3. Results, Students with science-based prior degrees performed better in the SBA assessments. The same occurred initially in the MEQs, but the effect diminished with time. The HP students performed consistently better but converged with other groups over time, particularly in the MEQs. Relative performance by the NONS students improved with time in both assessment formats. Overall, differences between the highest and lowest groups were small and very few students failed to meet the overall standard for the summative assessments. HP and BMS students had the lowest failure rate. NONS students were more likely to fail the assessments in Year 2 and 3, but their pass rates were still high. Female students performed significantly better overall at the end of Year 2 and in Year 3. There were only minor differences between Australian resident and International students. Conclusion, While there are small differences in performance in B & CS early in the programme, these lessen with time. The study results will inform decisions regarding timing of summative assessments, selection policy and for providing additional support to students who need it to minimize their risk of failure. Readers should note that this paper refers to student performance in only one of the four curriculum themes, where health professional and science graduates would be expected to have a significant advantage. [source] Evaluation of the Clinical Anatomy Program in the Medical School of Porto by two cohorts of studentsCLINICAL ANATOMY, Issue 1 2002M.A.F. Tavares Abstract The discipline of Clinical Anatomy, as introduced in the Medical School of Porto in academic year 1995/96, involved major changes in the way we teach anatomy to medical students, by adopting a clinically oriented approach. A questionnaire was designed to evaluate the opinion of second-year medical students enrolled in the program concerning main aspects of the discipline in two consecutive years; 84% of the students returned the questionnaire in 1996/97, and 70% in 1997/98. Students were asked about the level of their approval of the organization of the discipline, the role of the teaching staff, lectures, practical sessions, educational media, and continuous and summative assessments. For items replicated in both academic years, the means of the sum of scores in each year were compared (Student's t -distribution). Whenever a significant difference was found, changes in individual items were tested (chi-square distribution). The evaluation of the discipline in each of the two years was highly favorable for most of the parameters analyzed. Clin. Anat. 15:56,61, 2002. © 2002 Wiley-Liss, Inc. [source] | ||||||||||