Enlarged ventricles are a marker of several brain diseases; however, they are also associated with normal aging. Better understanding of the distribution of ventricular sizes in a large population would be of great clinical value to robustly define imaging markers that distinguish health and disease. The AGES-Reykjavik study includes magnetic resonance imaging scans of 4811 individuals from an elderly Icelandic population. Automated brain segmentation algorithms are necessary to analyze such a large data set but state-of-the-art algorithms often require long processing times or depend on large manually annotated data sets when based on deep learning approaches. In an effort to increase robustness, decrease processing time, and avoid tedious manual delineations, we selected 60 subjects with a large range of ventricle sizes and generated training labels using an automated whole brain segmentation algorithm designed for brains with ventriculomegaly. Lesion labels were added to the training labels, which were subsequently used to train a patch-based three-dimensional U-net Convolutional Neural Network for very fast and robust labeling of the remaining subjects. Comparisons with ground truth manual labels demonstrate that the proposed method yields robust segmentation and labeling of the four main sub-compartments of the ventricular system.