Human Activity Recognition based on Skeleton and Point Cloud Data

Abstract

With the demand for understanding human actions automatically, Human Activity Recognition is now becoming a prominent topic in the area of artificial intelligence. Human activity can be represented by multiple sources of data, such as RGB, optical flow, skeleton, depth, and point cloud. These various data types from different sensors enable HAR applications in numerous real-life scenarios, including surveillance, healthcare, and human-computer interaction. Among all kinds of data, skeleton and point cloud data have shown great potential in real-life applications. Researchers have focused on developing methods that combine the advantages of these two types of data. Skeleton data is effective in capturing the topological graph of human poses and offers computational efficiency compared to more complex data types. Point cloud data, on the other hand, has its advantages for indoor activities and information collection within a fixed space. We can first collect point cloud data and convert it into skeleton data, which has similar properties, for further processing. We propose a framework called Skeletal Dynamic Graph Convolutional Neural Network that first utilizes Dynamic Graph Convolutional Neural Network to convert raw point cloud data into skeleton data, preserving useful feature information for classification in the next stage. Then, the network, Spatial Temporal Graph Convolutional Network++, is applied for classification tasks to process the converted skeleton data. We validate our framework on the MMActivity dataset and the DGUHA dataset. Our model outperforms on the MMActivity dataset and the DGUHA dataset with Top-1 accuracy of 99.73% and 99.25%, and F1-Score of 99.62% and 99.25% respectively.