Finding feasible solutions and a good approximate Pareto front (PF) are two important tasks in the constrained multi-objective optimization (CMO). Various constraint handling techniques (CHTs) have a significant impact on these two tasks. To realize the adaptive adjustment of CHTs, a constrained multi-objective differential evolution algorithm with adaptive constraint handling technique (ACHT-CMODE) is proposed in the current study. In the ACHT-CMODE, three state-of-the-art constraint handling methods are integrated and an improved reverse generation distance is used to evaluate their performances. Also, the Q-learning method is utilized to guide the evolution of CHTs. The performance of the ACHT-CMODE is compared with that of the other five constraint multi-objective evolutionary algorithms on 18 test functions. Experimental results show that the overall performance of the ACHT-CMODE is the best among all compared algorithms, and the proposed algorithm is capable of selecting a suitable constraint handling method to solve a particular type of constrained multi-objective optimization problems (CMOPs).