The task of recognizing human faces from frontal views with expressions, illumination changes, and occlusions had been in investigated deeply by many proposed algorithms. However, few researches are focused on the problem of recognizing human faces with varying pose angles. For this problem, based on the usage of local descriptors, we propose a face recognition system that mainly consists of weighting face subjects from a feature’s view and consideration to the deformation degree between faces. For weighting face subjects from a feature’s view, it provides a more precise matching in local descriptors than Nearest NNNDR, a popular matching strategy. Considering the deformation degree between faces gives a new insight into faces with varying pose angles. In the recognition system, we use the technique of facial features localization to assist in finding the vicinity of a feature and measuring the deformation degree between faces. The face recognition system is experimented on the AR database and the FERET database. We use the support rate of the answer subject, the rate of no face detection, and the recognition rate to observe the behavior of our system. In the experiments, the support rate of the answer subject increases significantly and a correlation between these three indices is found. The recognition rate of our method is 97.49% for faces with a pose angle within ±40 degrees and without occlusions. We also discuss the impact of occlude faces with varying pose angles.