The buildings in Taiwan during the Japanese colonial period were very unique architectural style. This study investigated the tourists' heritage image of the architecture, and their revisiting willingness. This study used a questionnaire to obtain 313 valid questionnaires. The results showed that the heritage image can be divided into the four factors of the cultural orientation, specialty orientation, art orientation, and environmental orientation. Tourist's socio-economic background and travel characteristics had a direct impact on the heritage image: but revisiting willingness did not directly affect the relationship. In terms of logistic regression analysis of the heritage image of tourists revisiting willingness, it found that more cultural orientation image, the higher of the willingness to revisit. It is necessarily to construct the visitors' image of cultural orientation, to improve their willingness to revisit. Comparatively, it can drive the value of cultural heritage in destination tourism.