In recent years, using of Microsoft's Kinect depth sensor to calculate the human skeleton is called depth skeleton detection. So the way of somatosensory detection can more diverse. Related researches are constantly raised and follow-up, and the action of the detection and analysis is also discussed. There are some motions cannot be detected by the skeleton from depth image, for example, foot-cross and lying will cause judgment failed. In order to improve this misjudgment, this thesis uses the centroid of the color regions to achieve the motion detection. Ensuring the tracking point is the skeleton of the body to achieve the color tracking and solving the problem of misjudgment and tracking. Let the skeleton be tracked correctly.