The objective of this study to minimize the mast deformation by structural optimization of the mast of rotary drilling rig in order to better control its functional operation and improve its performance. For this purpose, a novel optimization scheme for the mast of rotary drilling rig is formulated using 3D graphics software Pro/E together with Finite Element Method. Back sliding wheel installed on the mast is firstly proved to reduce the deformation of the mast of rotary drilling rig effectively. The position of back sliding wheel relative to the mast of rotary drilling rig is adjusted, together with the position of pulley yoke, to obtain the minimum deformation of the mast of rotary drilling rig, which improves the stability of the mast of rotary drilling rig. Optimization efficiency and computational accuracy are demonstrated numerically.