The current procedure for diagnosis of Crohn's disease (CD) from Capsule Endoscopy is a tedious manual process which requires the clinician to visually inspect large video sequences for matching and categorization of diseased areas (lesions). Automated methods for matching and classification can help improve this process by reducing diagnosis time and improving consistency of categorization. In this paper, we propose a novel SVM-based similarity learning method for distinguishing between correct and incorrect matches in Capsule Endoscopy (CE). We also show that this can be used in conjunction with a voting scheme to categorize lesion images. Results show that our methods outperform standard classifiers in discriminating similar from dissimilar lesion images, as well as in lesion categorization. We also show that our methods drastically reduce the complexity (training time) by requiring only one half of the data for training, without compromising the accuracy of the classifier.