Abstract Due to the wide use of nonlinear loads, there are many serious problems of harmonics in the power system. It may cause voltage distortion, power quality degradation and protective relays malfunction. Therefore, the effect analysis of harmonics on the protective relays is necessary. In order to made the protective relays function properly under harmonic effects, this paper examines the small-energy overcurrent relay characteristics using experiment. Based upon the measurement results, some suggestions for improvement on relay setting are proposed. The test results may provide a valuable reference in applying and designing on protective schemes of relays.