This study measured the acceleration by consumer's smartphones in the buildings. Once the smartphones were triggered by an excitation, an embedded artificial neural network (ANN) model would classify whether the event was an earthquake or not within a few seconds. If the triggered motion was considered an earthquake event, then smartphones will record the whole vibration history of this event. After the earthquake event finished, the smartphones used a beacon micro-location technique to collect the location of the smartphones in the buildings more precisely and the relevant building information. Furthermore, the smartphones exchanged the vibration data with the smartphones on adjacent floors by Wi-Fi direct. Using the exchanged data, the smartphones could calculate interstory drifts on each floor. By comparing the maximum interstory drift ratio (IDR) to the preassigned threshold, the possible damage level of buildings during the earthquake could be estimated. However, the orientation of the smartphones is arbitrary and unknown. In addition, each smartphone's clock drifts with time. Therefore, this study dedicated to reducing the errors of the calculated interstory drift by employing signal process techniques and network time protocol (NTP) techniques to correct the orientations and to synchronize the clocks, respectively. Shaking table tests of a four-story steel building were conducted to verify the proposed methods. The results showed that the accuracy of damage detection of the building is 92%. Therefore, the proposed approach is quite feasible.