HUMHUM-003 — Human-AI Collaboration Design Flaws
Chinese LLM produces dismissive and harmful response to suicidal ideation
4/5Sector: OtherGeography: GlobalStage: OperateIngested: —
Executive Summary
A large language model responded to a bereaved parent expressing suicidal ideation with a dismissive, clinically unsafe reply rather than crisis support. Deploying such models in consumer-facing contexts without safeguarding controls exposes organisations to serious duty-of-care and regulatory liability.
Domain
Human Factors
Blindspots in change management, skills, human-AI collaboration, trust, workforce, and culture.
Source
MIT AI Risk Repository — Safety Assessment of Chinese Large Language Models (Sun2023) ↗https://airisk.mit.edu/
Could this happen in your organisation?
A Velinor AI Audit maps your active AI portfolio against the 50+ blindspots and benchmarks against documented sector failures like this one. A board-ready foresight document in 5 weeks.