Alzheimer's disease (AD) and Mild Cognitive Impairment (MCI) are characterized by widespread pathological changes in the brain. At the same time, Alzheimer's disease is heritable with complex genetic underpinnings that may influence the timing of the related pathological changes in the brain and can affect the progression from MCI to AD. In this paper, we present a multivariate imaging genetics approach for prediction of conversion to Alzheimer's disease in patients with mild cognitive impairment. We employ multivariate pattern recognition approaches to obtain neuroimaging and polygenic discriminators between the healthy individuals and AD patients. We then design, in a linear manner, a composite imaging-genetic score for prediction of conversion to Alzheimer's disease in patients with mild cognitive impairment. We apply our approach within the Alzheimer's Disease Neuroimaging Initiative and show that the integration of polygenic and neuroimaging information improves prediction of conversion to AD.