The driving experience has improved significantly due to the advancement of automotive information and electronic technologies. Vehicles are becoming intelligent and connected, which are named Intelligent Connected Vehicles (ICVs). However, the security problems of ICVs have emerged, and the cyber security of ICVs has become one of the most popular research fields to avoid ICVs being invaded. To mitigate the security threats of ICVs and enhance their defense ability, we investigate ICV security from the aspects of vehicle attacks and defenses. We classify the attack and defense technologies in terms of the functions of various information subsystems in the ICV, i.e., the ICV information perception system, in-vehicle communication system, control system, and V2X communication systems. We further highlight the security challenges and potential research directions to facilitate the development of better security mechanisms for ICV security.