Optical scatterometry is crucial to advanced nodes due to its ability of non-destructively and rapidly retrieving accurate 3D profile information. In recent years, an angle-resolved polarized reflectometry-based scatterometry which can measure critical dimensions, overlay, and focus in single shot has been developed. In principle, a microscope objective collects diffracted light, and pupil images are collected by a detector. For its application of calibrating lithography models, the pupil images are fit to a database pre-characterized usually by rigorous electromagnetic simulation to estimate dimensional parameters of developed resist profiles. The estimated dimensional parameters can then be used for lithography model calibration. In this work, we propose a new method to calibrate lithography models without needing dimensional parameter estimation. This method can be used to calibrate both rigorous physical models for process and equipment development and monitoring, and fast kernel-based models for full-chip proximity effect simulation and correction.