TY - GEN
T1 - Dependable Neural Networks for Safety Critical Tasks
AU - O’Brien, Molly
AU - Goble, William
AU - Hager, Greg
AU - Bukowski, Julia
N1 - Publisher Copyright:
© 2020, Springer Nature Switzerland AG.
PY - 2020
Y1 - 2020
N2 - Neural Networks are being integrated into safety critical systems, e.g., perception systems for autonomous vehicles, which require trained networks to perform safely in novel scenarios. It is challenging to verify neural networks because their decisions are not explainable, they cannot be exhaustively tested, and finite test samples cannot capture the variation across all operating conditions. Existing work seeks to train models robust to new scenarios via domain adaptation, style transfer, or few-shot learning. But these techniques fail to predict how a trained model will perform when the operating conditions differ from the testing conditions. We propose a metric, Machine Learning (ML) Dependability, that measures the network’s probability of success in specified operating conditions which need not be the testing conditions. In addition, we propose the metrics Task Undependability and Harmful Undependability to distinguish network failures by their consequences. We evaluate the performance of a Neural Network agent trained using Reinforcement Learning in a simulated robot manipulation task. Our results demonstrate that we can accurately predict the ML Dependability, Task Undependability, and Harmful Undependability for operating conditions that are significantly different from the testing conditions. Finally, we design a Safety Function, using harmful failures identified during testing, that reduces harmful failures, in one example, by a factor of 700 while maintaining a high probability of success.
AB - Neural Networks are being integrated into safety critical systems, e.g., perception systems for autonomous vehicles, which require trained networks to perform safely in novel scenarios. It is challenging to verify neural networks because their decisions are not explainable, they cannot be exhaustively tested, and finite test samples cannot capture the variation across all operating conditions. Existing work seeks to train models robust to new scenarios via domain adaptation, style transfer, or few-shot learning. But these techniques fail to predict how a trained model will perform when the operating conditions differ from the testing conditions. We propose a metric, Machine Learning (ML) Dependability, that measures the network’s probability of success in specified operating conditions which need not be the testing conditions. In addition, we propose the metrics Task Undependability and Harmful Undependability to distinguish network failures by their consequences. We evaluate the performance of a Neural Network agent trained using Reinforcement Learning in a simulated robot manipulation task. Our results demonstrate that we can accurately predict the ML Dependability, Task Undependability, and Harmful Undependability for operating conditions that are significantly different from the testing conditions. Finally, we design a Safety Function, using harmful failures identified during testing, that reduces harmful failures, in one example, by a factor of 700 while maintaining a high probability of success.
KW - Machine learning testing and quality
KW - Neural network dependability
KW - Neural network safety
KW - Reinforcement Learning
UR - http://www.scopus.com/inward/record.url?scp=85097389444&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85097389444&partnerID=8YFLogxK
U2 - 10.1007/978-3-030-62144-5_10
DO - 10.1007/978-3-030-62144-5_10
M3 - Conference contribution
AN - SCOPUS:85097389444
SN - 9783030621438
T3 - Communications in Computer and Information Science
SP - 126
EP - 140
BT - Engineering Dependable and Secure Machine Learning Systems - Third International Workshop, EDSMLS 2020, Revised Selected Papers
A2 - Shehory, Onn
A2 - Farchi, Eitan
A2 - Barash, Guy
PB - Springer Science and Business Media Deutschland GmbH
T2 - 3rd International Workshop on Engineering Dependable and Secure Machine Learning Systems, EDSMLS 2020
Y2 - 7 February 2020 through 7 February 2020
ER -