This paper presents an analytical and experimental study on the belt vibration and instability of a toothed-belt drive. The free span of the drive is considered as an axially moving beam with a periodically varying shape. The equations of motion, governing the static equilibrium and dynamic behavior of the free-span in the transverse direction, are derived by employing a variational principle. The geometrical boundary conditions are determined from the physical configuration of the band/wheel system. The natural frequencies vary as the belt tension and transporting velocity change. Instability of the toothed-belt drive can be divided into two aspects: divergence and parametric instability. Divergence instability may be generated by the Coriolis acceleration when the belt speed exceeds a critical value. Parametric instability can occur due to the periodic variations of belt shape and belt tension. A perturbation method is adopted to predict the stability of the toothed-belt drive. Attention is paid especially to the effect of the periodically distributed teeth of an axially moving belt. By comparing with the experimental results, favorable results in the predictions of natural frequencies are obtained.