The pursuit of larger data volume and higher switching speed in networking has emerges the 4-level pulse-amplitude modulation (PAM4) for the SerDes systems. In this thesis, the worst-case bits pattern method is proposed to derive the worst-case eye diagram for PAM4 without large PRBS simulation. Then, an optimization algorithm is established to search for the best finite impulse response (FIR) filter coefficients based on PAM4 extended peak distortion analysis (PDA) to improve PAM4 eye diagram efficiently. Finally, it is applied to calculate the suitable layout region for NRZ and PAM4 respectively, thereby addressing the preference zone for NRZ and PAM4.