Modulation instability (MI) is one of the mechanisms which results in pattern formation in the nonlinear wave system. While an incident wave with small amplitude perturbation passes through the nonlinear materials, some spatiotemporal frequencies may have the higher growth rate and grows up to form patterns. For MI to happen, the Lighthill criterion saying that the nonlinearity of material works in the opposite direction to the dispersion during the wave propagation must be satisfied. In optics, therefore MI is believed traditionally to exist in the self-focusing medium. In this dissertation, we study and present the characteristic behaviors of MI in the non-instantaneous self-defocusing medium. In Chapter One, we first overview the formation of MI and introduce some fundamental backgrounds, such as nonlinearity and non-instantaneity of the material response. Then in Chapter Two, we demonstrate theoretically and experimentally that MI can exist in the self-defocusing medium with the growth rate being the function of the spatial and the temporal frequency if the non-instantaneity is taken into consideration. Furthermore, we put such system into a ring cavity in Chapter Three. Interestingly, we find that spontaneous MI pattern can happen if the amount of feedback exceeds a certain threshold and the corresponding spatiotemporal frequency of the pattern satisfies the phase matching condition similar to that of a laser. Then in Chapter Four, we try to apply such MI in non-instantaneous media to achieve slow light, since MI contributes not only to the growth of amplitude but also to the phase of the wave. Finally, we do a conclusion in Chapter Five.