Recent expansion in medical education has substantially increased the supply of physicians. The literature has documented that a greater supply of physicians may not reduce the inequality of physician distribution measured by the Gini Coefficient or other single measures of inequality. However, more careful evaluation is required to determine the keys to addressing the allocation problems. We use a quantile regression with a spatial lag setup to investigate the distributional patterns of physician density among townships in Taiwan between 2010 and 2018. We find that the increased total number of physicians improved the physician density more in the townships on the lower quantiles than elsewhere in the distribution. This implies that physicians have been allocated more to the needier areas and this therefore improves the access to health care in these areas even though the inequality measured by the Gini coefficient gets worse. In general, the share of the aged population, median income, and whether a place has a medical center have greater and positive impacts on physician density in the higher quantiles. On the contrary, having a regional hospital generates a larger positive impact in the lower quantiles. The results suggest that regional hospitals have greater policy implications than medical centers in improving the allocation of physician services.