Target detection and tracking is one of the most important and fundamental technologies for a machine vision system. The purpose of this paper is to design a stereo vision-based dual-target tracking system. The system consists of two image tracking platforms, where one platform includes two cameras and two rotational degrees of freedom, can track their moving target in live video streaming. To begin with, we use color recognition and image processing methods to detect moving targets with different colors in continuous image streaming. Next, after the process of image labeling and filtering, the centroid of each target in image sequences will be determined. Finally, we can estimate the relative position between targets and cameras by stereo vision algorithm. With converting the relative position into motor control signals, the two image tracking platforms will be able to track their moving targets.