Abstract This dissertation is mainly to present some interesting properties of the Modulation Instability(MI) in the non-instantaneous self-focusing incoherent cavity. Modulation instability, happening in many nonlinear wave systems, is a phenomenon that a small amplitude modulation of a carrier wave exponentially grows due to the nonlinear response of the medium. To form MI, the Lighthill criterion should be satisfied, that is, the non-linearity and the dispersion (diffraction) must work oppositely. Through theoretical analysis and computer simulation, we studied the evolution of MI patterns in a nonlinear cavity which is longer than the coherence length of the light circulating in it. The patterns exhibit spectral line narrowing as the feedback is increased, resembling the line narrowing in lasers. In chapter 1, we provide an introduction to the modulation instability and some important properties – non-linearity and non-instantaneity – which play important roles in our analysis and simulation. The theoretical analysis of the pattern formation in the optical cavity is discussed in chapter 2, and the intuition behind that is presented. Chapter 3 is devoted to the simulation methods and some results of our system. Finally, we give a summary and the future work in chapter 4.