With the rapid development of information [1] technology and the rapid rise of sensor field, more and more ways and means for human beings to obtain information. The source or form of each information can be called a modality. For example, people have touch, hearing, vision, smell, information media, voice, video, text, etc., a variety of sensors, such as radar [2], infrared, accelerometers and so on. Each of these can be called a modality. In view of the problem that the recognition degree of single mode in human activity recognition is not high and the recognition rate is low, the fusion of multimodals will be applied to human activity recognition, which will greatly improve the accuracy of recognition and the efficiency of recognition. In this paper, the multimodal fusion algorithm is used to identify the wiSDM data set in the direction of human activity [3].