In this study, the research goal is to analyze the traffic flow with the vision-based method. However, the vision-based methods may face the problems of serious illumination variation, shadows, moving clouds, or swaying trees. Here, we propose a novel vehicle detection method without background modeling to overcome the aforementioned problems. First, a modified block-based frame differential method is established to quickly detect the moving targets without the influences of rapid illumination changes. Second, the precise targets' regions are extracted by the dual foregrounds fusion method. Third, a texture-based object segmentation method is proposed to segment each vehicle from the merged foreground image blob and remove the shadows. Fourth, a false foreground filtering method is developed based on the concept of motion entropy to remove the false object regions caused by the swaying trees or moving clouds. Finally, the texture-based target tracking method is proposed to track each detected target and then apply the virtual-loop detector to compute the traffic flow. Furthermore, the traffic jam event is also detected in the proposed system. Experimental results show that our proposed system can work in real time with the computing rate above 20 fps and the average of accuracy of vehicle counting can approach 86%.