Penalty functions are often used to handle constrained optimization problems in evolutionary algorithms. However, most of the penalty techniques use the same penalty coefficient in all constraints. In this paper, we introduce the rough set theory as a novel penalty adjustment method by considering the violation degree of each constraint. For numeric optimization problems, experimental results show that the proposed algorithm can achieve better solutions than several existing algorithms in constrained optimization problems. The results also show that the proposed rough penalty methods further improve the exploration and exploitation abilities. Furthermore, we apply this algorithm to deal with the multicast routing problem and show that the proposed algorithm is both effective and efficient.