The microstructures of polymer phase in polymer modified asphalt (PMA) were obtained with the introduction of fluorescence optical microscopy. To make quantitative analysis on these microstructures, the microscopy was equipped with hardware and image analysis software. Softening point was chosen as the representative of rheological properties of PMA. Several key factors affecting the rheological properties of PMA were discussed. According to the Krieger-Dougherty (KD) model , zero shear viscosity of PMAs (η0,PMA) is mainly influenced by zero shear viscosity of base asphalt (η0,A) volume fraction of polymer phase (φp),and maximum liquid fraction (φm), Relationship between η0, PMA and microstructure was fully described in detail by the KD model. Factors affectingφp and φm were also discussed. Through analyzing the particle size, shape, and distribution of polymer phase in PMA, we can evaluate the mechanical properties of the mix effectively. At last, the relationship between rheological properties and microstructure of PMA was discussed and determined in this paper.