The cerebellum plays an important role in both motor control and cognitive functions. Several methods to automatically segment different regions of the cerebellum have been recently proposed. Usually, the performance of the segmentation algorithms is evaluated by comparing with expert delineations. However, this is a laboratory approach and is not applicable in real scenarios where expert delineations are not available. In this paper, we propose a method that can automatically detect cerebellar lobule segmentation outliers. Instead of only evaluating the final segmentation result, the intermediate output of each segmentation step is evaluated and considered using a Hidden Markov Model (HMM) to produce a global segmentation assessment. For each intermediate step, a state-of-the-art image classification model ag-of-Words (BoW) is applied to quantize features of segmentation results, which then serves as observations of the trained HMM. Experiments show that the proposed method achieves both a high accuracy on predicting Dice of upcoming segmentation steps, and a high sensitivity to outlier detection.