Meta is enhancing its safety features on Instagram to alert parents if their teenage children discuss self-harm or suicide with the platform’s AI chatbots. The new measures, launching Thursday in Canada, the United States, Britain, and Australia, are designed to provide earlier intervention for at-risk youth and supplement crisis resources already offered by the company.
Under the updated system, parents who have activated Instagram’s optional parental supervision tools will receive notifications if the AI detects that their teen may be expressing intentions of self-harm or suicidal thoughts during conversations with the chatbot. Meta said it has developed a specialized AI system to identify such discussions, relying on subtle cues informed by consultations with dozens of mental health experts. Each flagged interaction will be reviewed by a human moderator before an alert is sent to caregivers.
If a teen’s statements are ambiguous, Meta has committed to a precautionary approach, alerting parents to ensure timely support, even at the risk of false positives. Parents using Instagram’s supervision features can also customize restrictions on the types of AI conversations their children are allowed to engage in, including barring discussion of a wider range of sensitive topics. Teens must consent or request to be supervised for the parental controls to take effect.
For teenagers who do not have parental supervision enabled, Meta said its AI will continue directing users toward crisis helplines and encourage seeking help from trusted adults. Additionally, should AI conversations suggest imminent risk, Meta has protocols in place to notify emergency services. In 2023, the company reported it made more than 19,000 referrals worldwide to first responders based on suicide risk assessments from posts on its platforms.
These announcements coincide with the recently enacted federal Safe Social Media Act (Bill C-34), which includes regulations aimed at improving safety on social media and AI platforms. The legislation mandates that AI chatbots operate responsibly, prohibiting them from encouraging self-harm or criminal activity. Bill C-34 also requires companies to implement crisis intervention protocols and to disclose their thresholds for alerting law enforcement about users posing potential harm to themselves or others.
Meta affirmed that its AI models for teen accounts remain subject to default content restrictions appropriate for users 13 years and older, and that the chatbot is programmed to avoid engaging in sexual or romantic topics or providing inappropriate content such as alcohol-related information. The company previously introduced stricter content settings for supervised teen accounts in October 2023, and these will now extend to interactions with AI chatbots.
Through collaboration with mental health professionals and parents, Meta aims to refine its AI’s sensitivity in identifying vulnerable young users and to equip parents with resources to navigate difficult conversations prompted by these alerts. The company acknowledged the limitations of automated detection and stressed ongoing efforts to balance proactive intervention with privacy considerations.