Comparative analysis of exit interviews and direct clinical observations in Pediatric Ambulatory Care Services in Afghanistan

Objective: To assess the receiver operating curves (ROCs) for counseling in the management of common childhood diseases comparing direct observations with exit interviews. Design: Eight thousand six hundred and fifty-nine randomly selected new outpatient consultations of sick children under 5 years were assessed by observation using a standardized checklist and an exit interview with their parent/guardian, taken between 2005 and 2007 from 948 health facilities in Afghanistan. The observation checklist was used as a 'gold standard' for counseling provided. Main measures: Sensitivity, specificity and ROCs were estimated for five counseling items, including explanations of: a working diagnosis; what to do at home; possible adverse reactions to medicine; signs that require a return to the health facility; and a time to return. Results: The prevalence of counseling items was relatively low (ranging from 8 to 80%), but generally increasing each year. Exit interviews had relatively low levels of sensitivity for the counseling items, ranging from 33 to 88%, with higher levels of specificity (ranging from 63 to 91%), whereas the ROCs ranged from 61 to 77%. Although ROCs varied significantly from year to year (P<0.002 for each item), there was little difference based on the sex or type of the health provider. Conclusions: Exit interviews did not provide reliable measurements of provider performance compared with direct observations. Observations identified low prevalence of counseling tasks though increasing over time. The differences between observation and exit interviews identified significant gaps in communication, suggesting that exit interviews are of low accuracy and should not be used alone.