A nonlinear mathematical model was utilized to investigate the behavior of surf-riding and broaching-to of a vessel in severe following seas. Based on the dynamics of six-degree-of-freedom rigid body motion, the model was derived with the consideration of wave force, hydrodynamic force, propeller and rudder force. An automatic control system was incorporated in the analysis so that the vessel could persist in her heading direction. Simulation was carried out for a trawler, and the critical initial surge velocity and the critical initial heading angle occurring surf-riding and broaching-to were predicted for different sea conditions. Furthermore, comparisons with a model experiment showed a fairly good agreement.