In the late Ming, the wide activities of the "mountain men" [or hermits] were already being criticized in terms of, "mountain men of the past were men in the mountains, but mountain men these days are men beyond the mountains" (Mingshi jishi benmo, juan 66, "Donglin dang yi"). The origins of the concept of mountain men beyond the mountains can be traced back to Beishan yiwen by Kong Zhigui (Wenxuan, juan 43), and the beginnings of their activities can be seen in the Tang dynasty. In the "Yin yi" chapter of Wenyuan yinghua, the mountain men were mainly occupied with selling herbal medicine; they shuttled in and out of the mountains. Li Bai called himself mountain man; Du Fu's neighbor, of the Chengdu Herbal Hall, was also a mountain man. This essay will comprehensively examine the activities of Tang dynasty mountain men as well as their connection with literary arts. Moreover, it discusses how this fashion for the custom of retirement from the world as hermits was propagated in Japan and Korea, and goes on to investigate how modern East Asian intellectuals transformed retirement from the world as a hermit from having a locus outside the world to having a this-worldly locus, as well as its significance.