Along with the progress of computer computation capabilities, sophisticated image processing/understanding methods have been developed and the functions of intelligent video surveillance systems have been greatly extended. In this thesis, we develop a video-based fire and smoke detection system based on the random forest algorithm. We use the distinct color and image variation properties of fire/smoke to select candidate regions. Then, image features of texture and motion patterns of the candidate regions are analyzed to determine any fire/smoke region. We propose to extract the features of both the texture and motion patterns of the fire/smoke with the local binary pattern (LBP) method. The random forest method is augmented to use the LBP features for fire/smoke detection to reduce false positive and enhance the fire and smoke detection rate.