While measuring the roundness of a work piece on a machine tool, the signal of a sensor includes the error of rotation axis and roundness error of work piece. By using the three-point method, one can separates the two kinds of error to get the roundness error of work piece. But in reality, there might be many errors of installation that would reduce the accuracy of the measurement. In this study, based on the theorem of three-points method measurement, we analyze the measuring error caused by errors of setting angle of the sensors. Then by using the optimum design method, we could find a set of setting angles of the sensors that would reduce the influence of the error of setting angles on measuring result. Numerical simulations as well as experiment are performed to validate the methodology.