This multicenter, retrospective study assessed the accuracy of machine algorithms in predicting suicidal behaviors using 53 studies from PubMed, PsycINFO, Scopus, EMBASE, IEEE, Medline, CINAHL, and Web of Science. Studies were included if machine learning algorithms assessed suicide or hospital-treated self-harm outcomes. A variety of model classes among these 53 studies were used, including 10 Random Forests, 8 Gradient-boosted trees, 5 ensemble learners, 5 LASSO, 3 Naive Bayes, and several natural language processing (NLP) models. The study compared the diagnostic accuracy of the AI models to real-world individual outcomes. This study found that the AI models assessed by the 53 studies demonstrated purposeful discrimination, with AUROC values ranging from 0.69 to 0.93. However, the study also recorded very low positive predictive values, ranging from 6% to 17%, indicating that among the AI models used in the 53 studies, greater than 80% of people who eventually died by suicide were missed by the high-risk flag, and conversely, greater than 80% of “high-risk” flags were false positives. As a synthesis, the paper undertook no temporal or external validation, as it extracted conclusions from many studies that did not perform any temporal or external validation. Furthermore, the study’s smaller sample size may have further inflated inaccuracies within the results.