Fish-eye lenses are common in several computer vision applications, such as four-camera surround view driver assistance, where a very wide angle (e.g., 180o) of view is available. Nevertheless, their applicability is usually limited by the lack of an accurate and easy-to-use calibration procedure. In this paper, we present a camera calibration method for fish-eye lenses and a panoramic image stitching framework for calibrated surround images. To achieve the calibration of fish-eye captured images, it only requires to observe a reference planar pattern once (e.g., chessboard) shown at a few different orientations, followed by offline performing homography estimation and save the related parameters. Each image can then be efficiently online transformed to the top-down view via the perspective transformation based on the estimated homography matrix. As a result, these surround images with top-down views can be stitched to generate the final panoramic image. It is expected that the proposed framework would be applicable to AVM (around view monitoring) system or ADAS (advanced driver assistance system) of vehicles in the future.