Continuity-structured clinical observations: Assessing the multiple-observer evaluation in a pediatric resident continuity clinic

OBJECTIVES. The continuity-structured clinical observation tool was developed and used to conduct a multiple-observer evaluation to assess residents in the core competency areas of patient care, interpersonal and communication skills, and professionalism. The objectives were to assess pediatric resident performance in a continuity clinic by using direct observation and to compare evaluations among preceptors, residents, and parents. METHODS. Pediatric residents in a large continuity clinic participated between August and December 2005. The continuity-structured clinical tool included items in the following domains: opening the interview, history taking, relationship skills, personal manner, negotiation or management, and physical examination. Each resident was directly observed during 1 entire patient encounter. Parents, preceptors, and residents completed evaluations by selecting 1 of 4 possible responses for performance of each item. We dichotomized responses as "yes" versus "no/partial" and analyzed aggregate scores for individual items and domains among the 3 evaluators by McNemar test, percentage agreement, and interoberserver agreement (κ). RESULTS. Fifty-four of 57 eligible residents had all 3 evaluations completed. Parents rated residents the highest and showed least variability (only 2 items for which parents indicated the task was completed in <90% of the encounters). Residents rated themselves the lowest (35 items with <90%). In comparing the residents and preceptors, the domains that had the lowest percentage of agreement were history taking (range: 61%-91%) and negotiation or management (range: 51%-88%). All of the evaluators scored residents the lowest in the domain of negotiation or management, with the following lowest-score items: probe for decision-makers, assess willingness and barriers, and use of visual aids. CONCLUSIONS. Compared with parents, residents and preceptors demonstrated greater variability in resident performance evaluations. All of the evaluators scored residents lowest in the domain of negotiation or management during continuity-clinic visits. Residency programs should strongly consider emphasizing skill development in this area.