In the construction of intelligent bus dispatching system, the problem of passenger congestion in buses has become an urgent problem. With the continuous development of computer vision technology and the gradual popularity of video surveillance systems in buses, it is possible to use video images in buses to detect congestion. Since the upper limit of the number of passengers on the bus is fixed, there is no need to predict the number of passengers with excessive passenger density. This paper chooses a unified multi-scale deep convolutional neural network with a relatively simple network structure to analyze bus passenger congestion. By using vgg16 as the density classification network for feature extraction, MSCNN is used to estimate the number of bus passengers and the definition and classification of the concept of congestion. According to the data set used in the experiment, the accuracy of predicting the congestion of passengers in the bus can reach 96.2%.The experimental results show that this method is of great significance to the analysis of bus passenger congestion.