This paper presents a video enhancement method for video surveillance applications, which achieves good enhancement quality and computational efficiency simultaneously. The proposed method effectively estimates the enhancement factor of each pixel by image downscaling based on spatial similarity. A low cost motion detection method is further integrated to maintain the enhancement state of each pixel based on temporal similarity. Experimental results show that the proposed method could save about 97% computational cost with acceptable quality for the outdoor benchmark PETs2001 and indoor benchmark Terrascope.