Collecting outcome data of a text messaging smoking cessation intervention with in-program text assessments: How reliable are the results?

Background: Text messaging interventions have shown promise in helping people quit smoking. Texting programs periodically survey participants about their smoking status. This study examined the consistency of participant self-reported smoking between external surveys and internal program text message assessments. Methods: Participants in Text2Quit program were surveyed about their past 7-day smoking at one, three, and six months post-enrollment using different survey modes (external surveys and internal program text message assessments) and responses were compared for consistency. The first set of analyses was conducted for participants responding on both modes (n = 45 at one month; n = 50 at three months; n = 42 at six months). Additional analyses, assuming missing = smoking, were conducted with the full sample of 262 smokers (68.7% female, mean age = 35.8 years) and compared to saliva-confirmed abstinence rates. Results: Participants responding to both modes consistently reported smoking status at one (88.9%), three (88.0%) and six (88.1%) months post-enrollment, with fair to substantial levels of agreement (one month: κ = 0.24; three months: κ = 0.63; six months: κ = 0.66). Participants responding to both modes reported high rates of abstinence. In missing = smoking analyses, significant differences in abstinence rates reported across modes were detected at each timepoint (one month: external = 30.5%, internal = 16.4%; three months: external = 33.2%, internal = 16.0%; six months: external = 31.7%, internal = 12.2%; all p <.001). Moderate levels of agreement were found between the two modes. At 6 months, abstinence rates obtained via internal data were closer to those biochemically verified (15.7%) compared to external surveys. Conclusions: Results provide initial support for the use of internal program assessments in text messaging programs with missing = smoking assumptions in order to gather outcome data on smoking behavior.