The purpose of this study is to propose a systematic approach for motion control of a novel ball screw transmission system, such that it can produce output motions with high accuracy First, the structures and characteristics of system are investigated. Then, an improving design is preformed to improve the precision of the mechanism. Moreover, the mathematical model of the system is obtained by the system identification experiment, and then the designs of the modified PID controller and the ILC controller are carried out. Furthermore, the correctness of the designs is validated by computer simulation. Finally, a prototype machine and an experimental setup are established and tested. The experimental results show that the experimental output agrees well with the required output speed. Consequently, the proposed method is feasible.