A good site layout planning can avoid possible conflicts of equipment and increase accessibility during a construction process. In current practice, it is difficult to re-plan layout during the site process since the location and quantity of on-site resources change over time. Many researchers have developed site cameras to facilitate dynamic space management. However, using cameras in the site still has fragmentation between site monitoring and planning. In this research, we aim to rapidly retrieve geometric information from site cameras for dynamic site planning. We developed a four-step method: projection, duplication, description and calibration. The first step, projection, is to establish the projective model of the camera between videos and actual site. The second step, duplication, is to rapidly acquire positions and dimensions of construction objects based on the video images and model their 3D geometry in geometric virtual construction. The third step, description, is to link the geometric models with the real objects and build semantic virtual constriction, which contains not only on-site scenarios but also engineers' knowledge. The fourth step, calibration, is to improve the accuracy of the virtual construction for realistic planning. We built a software tool by integrating the four steps. The tool allows engineer to load the video and specify the location of each object. They are also able to specify the meaning of the objects. The algorithms are simultaneously creating the numerical models to link the images on the video with the virtual models. We used a real campus building to validate the usability of our method. Using the CCTV videos retrieved from the site, it took 123 seconds to create a virtual construction site. We compared the actual objects with virtual ones and found the errors are from 0.2 to 1.2 meters. The results indicate our method is feasible to transfer video to numerical virtual construction site within a reasonable time and accuracy.