Magnetic flux leakage (MFL) inspection technique is widely used in the inspection and evaluation of ferromagnetic materials. Different processing methods and application environment will cause the difference of material surface roughness (Ra) which becomes one of the key factors influencing the detection signal during MFL inspection. This paper analyzed the industrial testing results with automated MFL equipment and comes cautiously into a conclusion that surface roughness has a major influence on testing results. In order to further study the surface roughness influence on MFL detection, this paper established a simplified model of the workpiece surface roughness. ANSYS simulation was used to analyze the background magnetic field caused by surface roughness and its impact on the resolution capacity of leakage magnetic field of surface defect. Furthermore, a method to improve signal noise ratio (SNR) based on the lift-off effect of sensors on rough surfaces is proposed.