Pitching segments are the starting points of every baseball event. Locating pitching segments accurately becomes a critical step in content analysis of baseball game video. However, existing scene detecting method either need complicate training process or labor effort labeling, and it might fail to deal with unseen data. In this paper, we present an unsupervised method to address the above problems. Given a video clip, the proposed method constructs clusters of video segments and ranks them to build a pitching model through four steps: video segment, segment clustering, clustering selection, and frame classification. The system which combines similarity analysis, Bayesian information criterion, and entropy for modeling and detecting pitching scene resolves the defect of existing methods. Our experiments also demonstrate a promising result of the proposed method.