This paper presents an image-based classification method, and applies it to classification of cardiac MRI scans of individuals with Tetralogy of Fallot (TOF). Clinicians frequently diagnose cardiac disease by measuring the ventricular volumes from cardiac MRI scans. Interrater variability is a common issue with these measurements. We address this issue by proposing a fully automatic approach for detecting structural changes in the heart. We first extract morphological features of each subject by registering cardiac MRI scans to a template. We then reduce the size of the features via a nonlinear manifold learning technique. These low dimensional features are then fed into nonlinear support vector machine classifier identifying if the subject of the scan is effected by the disease. We apply our approach to MRI scans of 12 normal controls and 22 TOF patients. Experimental result demonstrates that the method can correctly determine whether subject is normal control or TOF with 91% accuracy.