Object tracking is a challenging topic in computer vision. Local descriptors generated from distinctive image features can perform a robust and rapid object tracking. However, the feature descriptors are usually vectors in a high-dimensional space, causing the conventional linear search method inefficient. In this research, two algorithms are introduced. The first one is called “SURF”, which is designed to detect the scale-and-rotation-invariant feature in a image, and then generate descriptors for each feature point. The second one is the priority search method combined with the K-means tree. This algorithm can perform fast searching especially for high-dimensional data such as SURF descriptors. In the experiment part of this study, it is showed that the priority search method can raise the object tracking FPS for about 30%, in contrast to conventional linear search.