House price fluctuations not only affect people's living quality and living standards, but also affect the healthy development of the national economy and the harmony and stability of the society. Discussing the main influencing factors of house price fluctuations has always been a hot issue in the theoretical and practical circles. On the basis of constructing a model of the impact mechanism of income and interest rates on housing prices, the sample numbers of 35 large cities from 2008 to 2020 are selected, and the panel quantile regression model is used for empirical analysis. The results show that housing prices in my country's large and medium-sized cities are mainly driven by income rather than costs. Income is the main factor that affects housing price increases. Interest rate changes have no significant impact on housing prices. The effects of various influencing factors are significantly different at different quantile levels. The higher the level of the city, the greater the effect of income on housing prices, while the less affected by factors such as cost and per capita GDP. The research conclusion has certain reference value for different cities to adopt corresponding control policies according to their own characteristics.