Developing and validating a parser-based suicidality detection model in text-based mental health services

Abstract

Background: Advances in text-mining can potentially aid online text-based mental health services in detecting suicidality. However, false positive remains a challenge. Methods: Data of a free 24/7 online text-based counseling service in Hong Kong were used to develop a novel parser-based algorithm (PBSD) to detect suicidal ideation while minimizing false alarms. Sessions containing keywords related to suicidality were extracted (N = 1267). PBSD first applies a sentence parser to work out the grammatical structure of each sentence, including subject, object, dependent and modifier. Then a set of syntax rules were applied to judge if a flagged sentence is a true or false positive. Half of the sessions were randomly selected to train PBSD, the remaining were used as the test set. A standard keywords matching model was adopted as the baseline comparison. Accuracy and recall were reported to demonstrate models' performance. Results: Of the 1267 sessions, 585 (46.2 %) were false alarms. The false alarms were categorized into four types: negation-induced false alarms (NIFA; 14 %), subject-induced false alarms (SIFA; 19 %), tense-induced false alarms (TIFA; 30 %), and other types of false alarms (OTFA; 37 %). PBSD significantly outperforms the baseline keywords matching model on accuracy (0.68 vs 0.53, 28.3 %). It successfully amended 36.8 % (105/297) lexicon matching-caused false alarms. The reduction on recall was marginal (1 vs 0.96, 4 %). Conclusions: The proposed model significantly improves the use of lexicon-based method by reducing false alarms and improving the accuracy of suicidality detection. It can potentially reduce unnecessary panic and distraction caused by false alarms among frontline service-providers.