Evaluating the quality of HIV epidemiologic evidence for populations in the absence of a reliable sampling frame: a modified quality assessment tool

Background: Sampling frames rarely exist for key populations at highest risk for HIV, such as sex workers, men who have sex with men, people who use drugs, and transgender populations. Without reliable sampling frames, most data collection relies on non-probability sampling approaches including network-based methods (e.g. respondent driven sampling) and venue-based methods (e.g. time-location sampling). Quality of implementation and reporting of these studies is highly variable, making wide-ranging estimates often difficult to compare. Here, a modified quality assessment tool, Global.HIV Quality Assessment Tool for Data Generated through Non-Probability Sampling (GHQAT), was developed to evaluate the quality of HIV epidemiologic evidence generated using non-probability methods. Methods: The GHQAT assesses three main domains: study design, study implementation, and indicator-specific criteria(prevalence, incidence, HIV continuum of care, and population size estimates). The study design domain focuses primarily on the specification of the target and study populations. The study implementation domain is concerned with sampling implementation. Each indicator-specific section contains items relevant to that specific indicator. A random subset of 50 studies from a larger systematic review on epidemiologic data related to HIV and key populations was generated and reviewed using the GHQAT by two independent reviewers. Inter-rater reliability was assessed by calculating intraclass correlation coefficients for the scores assigned to study design, study implementation and each of the indicator-specific criteria. Agreement was categorized as poor(0.00–0.50), fair(0.51–0.70), and good(0.71–1.00). The distribution of good, fair, and poor scores for each section was described. Results: Overall, agreement between the two independent reviewers was good(ICC >0.7). Agreement was best for the section evaluating the HIV continuum of care(ICC = 0.96). For HIV incidence, perfect agreement was observed, but this is likely due to the small number of studies reviewed that assessed incidence(n = 3). Of the studies reviewed, 2% (n = 1) received a score of “poor” for study design, while 50% (n = 25) received a score of “poor” for study implementation. Conclusions: Addressing HIV prevention and treatment needs of key populations is increasingly understood to be central to HIV responses across HIV epidemic settings, though data characterizing specific needs remains highly variable with the least amount of information in the most stigmatizing settings. Here, we present an efficient tool to guide HIV prevention and treatment programs as well as epidemiological data collection by reliably synthesizing the quality of available non-probability based epidemiologic information for key populations. This tool may help shed light on how researchers may improve not only the implementation of, but also the reporting on their studies.