Magnetic resonance imaging (MRI) brain features have played a great role in the study of brain aging. However, the performance of MRI brain features in evaluating brain aging have not been clearly addressed. This study aimed to propose an optimized strategy of Relief scoring to evaluate the brain features by comparing their weights in the classification of brain aging. Three kinds of brain features, diffusion, morphology, and network, were first deduced from the multimodality images of TI-weighted imaging and diffusion tensor imaging. Then, significant brain features were extracted from deduced brain features using t tests. Further, four kinds of optimized brain features were chosen from significant features by Relief scoring. To validate the performance of optimized features, the support vector machine was implemented in the classification of brain aging by comparing optimized brain features with initial brain features. The results showed that Relief scoring demonstrated robust capability in feature selection for brain aging. The optimized brain features after Relief scoring had better performance than that without Relief scoring in the classification of brain aging. The brain features optimized by Relief scoring could be used to extract significant brain features for evaluating brain aging.