TY - JOUR
T1 - The Needs and Benefits of Continuous Model Updates on the Accuracy of RT-Induced Toxicity Prediction Models Within a Learning Health System
AU - Nakatsugawa, Minoru
AU - Cheng, Zhi
AU - Kiess, Ana
AU - Choflet, Amanda
AU - Bowers, Michael
AU - Utsunomiya, Kazuki
AU - Sugiyama, Shinya
AU - Wong, John
AU - Quon, Harry
AU - McNutt, Todd
N1 - Publisher Copyright:
© 2018 Elsevier Inc.
PY - 2019/2/1
Y1 - 2019/2/1
N2 - Purpose: Clinical data collection and development of outcome prediction models by machine learning can form the foundation for a learning health system offering precision radiation therapy. However, changes in clinical practice over time can affect the measures and patient outcomes and, hence, the collected data. We hypothesize that regular prediction model updates and continuous prospective data collection are important to prevent the degradation of a model's predication accuracy. Methods and Materials: Clinical and dosimetric data from head and neck patients receiving intensity modulated radiation therapy from 2008 to 2015 were prospectively collected as a routine clinical workflow and anonymized for this analysis. Prediction models for grade ≥2 xerostomia at 3 to 6 months of follow-up were developed by bivariate logistic regression using the dose-volume histogram of parotid and submandibular glands. A baseline prediction model was developed with a training data set from 2008 to 2009. The selected predictor variables and coefficients were updated by 4 different model updating methods. (A) The prediction model was updated by using only recent 2-year data and applied to patients in the following test year. (B) The model was updated by increasing the training data set yearly. (C) The model was updated by increasing the training data set on the condition that the area under the curve (AUC) of the recent test year was less than 0.6. (D) The model was not updated. The AUC of the test data set was compared among the 4 model updating methods. Results: Dose to parotid and submandibular glands and grade of xerostomia showed decreasing trends over the years (2008-2015, 297 patients; P <.001). The AUC of predicting grade ≥2 xerostomia for the initial training data set (2008-2009, 41 patients) was 0.6196. The AUC for the test data set (2010-2015, 256 patients) decreased to 0.5284 when the initial model was not updated (D). However, the AUC was significantly improved by model updates (A: 0.6164; B: 0.6084; P <.05). When the model was conditionally updated, the AUC was 0.6072 (C). Conclusions: Our preliminary results demonstrate that updating prediction models with prospective data collection is effective for maintaining the performance of xerostomia prediction. This suggests that a machine learning framework can handle the dynamic changes in a radiation oncology clinical practice and may be an important component for the construction of a learning health system.
AB - Purpose: Clinical data collection and development of outcome prediction models by machine learning can form the foundation for a learning health system offering precision radiation therapy. However, changes in clinical practice over time can affect the measures and patient outcomes and, hence, the collected data. We hypothesize that regular prediction model updates and continuous prospective data collection are important to prevent the degradation of a model's predication accuracy. Methods and Materials: Clinical and dosimetric data from head and neck patients receiving intensity modulated radiation therapy from 2008 to 2015 were prospectively collected as a routine clinical workflow and anonymized for this analysis. Prediction models for grade ≥2 xerostomia at 3 to 6 months of follow-up were developed by bivariate logistic regression using the dose-volume histogram of parotid and submandibular glands. A baseline prediction model was developed with a training data set from 2008 to 2009. The selected predictor variables and coefficients were updated by 4 different model updating methods. (A) The prediction model was updated by using only recent 2-year data and applied to patients in the following test year. (B) The model was updated by increasing the training data set yearly. (C) The model was updated by increasing the training data set on the condition that the area under the curve (AUC) of the recent test year was less than 0.6. (D) The model was not updated. The AUC of the test data set was compared among the 4 model updating methods. Results: Dose to parotid and submandibular glands and grade of xerostomia showed decreasing trends over the years (2008-2015, 297 patients; P <.001). The AUC of predicting grade ≥2 xerostomia for the initial training data set (2008-2009, 41 patients) was 0.6196. The AUC for the test data set (2010-2015, 256 patients) decreased to 0.5284 when the initial model was not updated (D). However, the AUC was significantly improved by model updates (A: 0.6164; B: 0.6084; P <.05). When the model was conditionally updated, the AUC was 0.6072 (C). Conclusions: Our preliminary results demonstrate that updating prediction models with prospective data collection is effective for maintaining the performance of xerostomia prediction. This suggests that a machine learning framework can handle the dynamic changes in a radiation oncology clinical practice and may be an important component for the construction of a learning health system.
UR - http://www.scopus.com/inward/record.url?scp=85058775255&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85058775255&partnerID=8YFLogxK
U2 - 10.1016/j.ijrobp.2018.09.038
DO - 10.1016/j.ijrobp.2018.09.038
M3 - Article
C2 - 30300689
AN - SCOPUS:85058775255
SN - 0360-3016
VL - 103
SP - 460
EP - 467
JO - International Journal of Radiation Oncology Biology Physics
JF - International Journal of Radiation Oncology Biology Physics
IS - 2
ER -