Last fall, OpenAI shared research showing that about 1 million people a week have conversations with ChatGPT that include explicit indicators of potential suicidal planning or intent.

In the sub-linked article:

our initial analysis estimates that around 0.15% of users active in a given week have conversations that include explicit indicators of potential suicidal planning or intent and 0.05% of messages contain explicit or implicit indicators of suicidal ideation or intent.

0.15% sounds rare on the surface, but percentages stop being intuitive at massive scales like that. If many people are using this thing as a suicide hotline, it needs to handle this gracefully; or at the very least not encourage them for God’s sake.