Accurate body appearance detections and 'non-radiation' X-ray synthesis using artificial intelligence and depth-sensing technologies

Abstract

Introduction: For impactful management of scoliosis, early detection and regular follow-up of the deformity is critical. Moiré topography and X-rays are commonly used methods. The former cannot accurately assess spine deformations and routine X-rays requires children to undergo increased ionising radiation exposure. Both methods are not portable and not equitably assessable across health systems. Objectives: We aim to develop and preliminarily validate a novel portable system for accurate body appearance detections and X-ray synthesis without radiation, enabling fast screening, out of hospital follow-up and accurate spine curvature predictions. The objectives include 1) automated surface anatomy landmarks detections; and 2) non-radiation X-rays syntheses. Materials and Methods: 892 patients attending our clinic were recruited prospectively. High-resolution optical image (RGB image) acquisition with concurrent depth-sensing (D image) were performed on all patients in standing upright posture and back facing the camera. Six surface landmarks including the inferior angles of the scapulae, the posterior superior iliac spines (PSIS), the 7th cervical vertebrae (C7), and the coccyx were labelled on the images of patient’s back by a spine specialist. This was annotated as the ground truth (GT) landmarks. The first 800 patient images were used to train (708) and validate (92) two deep learning networks. A High-Resolution Network (HRNet) was used for surface anatomy landmarks detection on the RGB images. Subsequently, a Wasserstein Generative Adversarial Network (WGAN) was implemented for X-ray synthesis guided by the 6 surface landmarks detected using HRNet. Thereafter, another dataset of 92 patients was used to test the performance of anatomy-landmark detection and the synthesized X-rays. The detected landmarks on the RGB-D images were quantitatively compared with the specialist’s GT measurements and the absolute distance error was calculated. The similarity between the synthesized X-ray and the original X-ray was evaluated using the multiscale-structural similarity index (MS-SSIM). All results were presented in mean ± SE. Results: The average age of the patients was 14 ± 0.1 with 75% being female. The Cobb angles ranged from 10⁰ to 105⁰ with an average value being 24 ± 0.4⁰ and 90% of patients had a Cobb angle smaller than 40⁰. The evaluation metric for the anatomy landmark detection accuracy indicated the mean absolute distance error was as small as 3.4 ± 0.5 pixels; demonstrating that the auto-detected anatomy landmarks were accurate. The mean MS-SSIM of the synthesized X-rays in comparison with the original X-rays was 0.703 ± 0.0125 (with 1 indicating 100% the same), indicating the images quality and structure of the synthesized X-rays were similar to the original X-rays. A close manual visual comparison of the synthesized X-rays was performed and examples are given in figure 1. Discussion and conclusion: The evaluation metrics suggested that RGBD integrated with deep learning is feasible in automatically detecting the key anatomy landmarks and generating valid synthetic X-rays with a portable acquisition system without radiation exposure. Objective assessments of the back appearance can be achieved with the auto-detected anatomy landmarks. Assisted with these auto-detected anatomy landmarks, the synthesized X-rays results were satisfied with a focus on the spine curvatures. The performance of the X-ray synthesis is better satisfied with a curve less than 40⁰ according to the visual assessments. This is because the dataset we collected to train the neural network had 90% patients with curves less than 40⁰. For future studies, the follow-up of these patients will be performed to test the ability of our method in accurately assessing curve progression