Application-level traffic classification is an essential requirement for stable network operation and resource management. The payload signature-based classifier is considered a reliable method for Internet traffic classification. However, with this system, processing speeds are slower when high volumes of traffic are being classified in high-speed networks in real time. In this paper, we propose a method for server IP-port pair cachebased traffic classification, with the aim of increasing the processing speed and completeness of payload signature-based traffic classification. This approach takes application traffic locality into consideration. Moreover, we propose a cache data management method that has the purpose of minimizing the utilization of cache memory and processing speed and maximizing level of completeness. When our proposed method was applied to a campus network, we observe 10 times improvement in processing speed and 10% increasing in completeness against the payload signature-based classifier without a server IP-Port pair cache.