Recently, the quantity of video information has increased at a high speed due to the improvement of the information technology. There are many methods for video retrieval. Similarity retrieval based on the contents is one of the suitable methods. In the video, the change of the spatial relationship between objects due to the movement of the object provides important information. In this paper, the spatial relationship between objects and its change are used as a base for determining if two videos are similar. In the mean time, the “implied relationship” is also considered in this paper for similarity retrieval of video. Finally, an example is given to demonstrate the process of the proposed method.