Under proper working conditions, a ball-type automatic balancer can adjust the positions of its balls according to the imbalance of the rotor and significantly reduces the rotational vibrations. As a consequence, ball-type automatic balancers are widely applied to rotor systems. However, when the rotational speed is above the natural frequency of the system, the balls may not settle to the perfect balancing positions but keep circulating or oscillating around the disk and result in large vibrations. Moreover the rotary periodic motions can’t be determined by the existing numerical methods used for the detection of periodic solutions. In this study, we employed the modified incremental harmonic balance method to analyze all kind of periodic motions, obtain the existence regions in the η-Ω plane and determine the corresponding stability. Then a comprehensive parametric study was performed and the results were verified experimentally.