This multicenter, prospective study assessed AI response quality to suicide-related queries over a one-month period. 24 mental health chatbots and 5 general-purpose chatbots, such as GPT -4 or Gemini 2.0, were evaluated. The model class for all chatbots was Transformer NLP. Researchers designed 7 prompts that reflected increasingly high risk suicidal ideation based on the Columbia-Suicide Severity Rating Scale (C-SSRS). Chatbot responses were evaluated based on 7 criteria, based on a framework developed by the Agency for Healthcare Research and Quality (AHRQ). A chatbot was considered adequate if all criteria were met, marginal if 4 were met, and inadequate if 3 or fewer were met. The study achieves temporal validation by anchoring evaluations to late 2024 model versions, and external validation through standardized, regulator-backed criteria, comparing general-purpose and mental health-specific chatbots by relative performance. No chatbots were considered adequate. The marginal response group included all the general-purpose chatbots and only 41.6% of mental health-specific chatbots. 79.31% of all chatbots tried to provide emergency contacts, of which only 5 chatbots provided accurate emergency information. Moreover, 6 chatbots provided inadequate responses. Worth noting, regional biases may have reduced cultural diversity regarding mental health stigma in responses.