The modal theory can be used to analyze the wave propagation in the linear and nonlinear planar optical waveguide. By solving the Maxwell's equation and matching the boundary conditions, we can find the eigenmode of transverse electric field in the planar optical waveguide with nonlinear guiding film. A detail modal analysis of the nonlinear waveguide structure is proposed. The analytical results are accompanied by numerical examples. The numerical simulation results show that the analyses are correct.