In order to study the dynamic characteristics of helical gear transmission systems, while considering factors such as constant driving torque excitation, constant load torque excitation, gear backlash, support bearings, support damping, and gravity, an 8‐degree of freedom nonlinear dynamic model of helical gear bearing bending torsion axis coupling was established using Lagrange's theorem. The fixed step Runge kutta method was used to numerically solve the model, and the vibration response sweep frequency map and vibration response time domain curve of the helical gear system considering time‐varying center distance under velocity excitation were obtained. The research results indicate that time‐varying center distance increases the vibration amplitude in the x direction and has a significant impact, while the vibration amplitude in the y and z directions decreases. As the velocity excitation increases, the vibration response amplitude of the system will increase.