Manifold learning using deep neural networks been shown to be an effective tool for noise reduction in low-dose CT. We propose a new iterative CT reconstruction algorithm, called Manifold Reconstruction of Differences (MRoD), which combines physical models with a data-driven prior. The MRoD algorithm involves estimating a manifold component, approximating common features among all patients, and the difference component which has the freedom to fit the measured data. While the manifold captures typical patient features (e.g. healthy anatomy), the difference image highlights patient-specific elements (e.g. pathology). We present a framework which combines trained manifold-based modules with physical modules. We present a simulation study using anthropomorphic lung data showing that the MRoD algorithm can both isolate differences between a particular patient and the typical distribution, but also provide significant noise reduction with relatively low bias.