With the increasing applications of Unmanned Surface Vehicles (USV), the complexity of navigable waters is also increasing. This paper proposes a navigation method using visual servo to accommodate outlier and loss of GPS signals near and under bridges so as to enhance the ability of USV to sail through the bridge safely. Firstly, the real-time scene of the bridge is collected through the visual system carried by the USV when preparing to sail through the bridge. Then the region of bridge aperture in the image is segmented by image preprocessing and threshold segmentation . With the extracted contour of the segmented region of the bridge aperture, the heading deviation angle and deviation distance are calculated and feed to the controller which adjusts the heading and speed of the USV in real time to drive the USV through the bridge safely. Finally, the performance of the proposed method is verified with real pictures taken by USV in actual environment. Simulation and experiment results show that the proposed algorithm achieves 96% accuracy rate for bridge aperture detection, and 100% safe driving rate when the starting point is within a distance of 4m from the center line of the bridge, which basically meets the requirements for safe sailing of the USV through the bridge.