Optimizing the temporal adjacency matrix for 3D human pose estimation through clustering

Abstract

With the booming development of multimedia technologies related to 3D images and videos, 3D human pose estimation has gained increasing attention. In 3D human pose estimation, exploring the temporal relationships between human joints is essential. Transformer-based methods for modeling temporal relationships leverage the temporal adjacency matrix within the self-attention mechanism as a key component for capturing these connections. We define temporal features that occupy a small portion of all temporal features and are distant from other temporal features in the high-dimensional space as noisy features. In the temporal adjacency matrix of self-attention, noisy features can interfere with the computation of other features, and thus their correlation with other features should be eliminated. However, existing methods overlook this issue. To address this issue, we propose a DBSCAN-based clustering module to detect noisy temporal features and an adjacency matrix masking mechanism to suppress their influence. First, we cluster the input temporal features to obtain noisy features and clustering results. Then, we eliminate the correlations of noisy features on other temporal features within the adjacency matrix and reduce the correlations between different classes. Extensive experiments on the Human3.6M and MPI-INF-3DHP datasets, using state-of-the-art methods as benchmarks, demonstrate that our approach achieves improvements of up to 6.94 % Mean Per Joint Position Error (MPJPE) compared to the original methods with ground–truth input.