HUMHUM-004 — Trust and Acceptance Issues
AI Companions Breach User Trust Through Unpredictable Failures
3/5Sector: OtherGeography: GlobalStage: OperateIngested: —
Executive Summary
Conversational AI that simulates emotional relationships periodically generates nonsensical or jarring outputs that shatter users' expectations of companionship or partnership. Organisations deploying such systems face reputational, psychological harm, and duty-of-care liabilities when vulnerable users experience betrayal.
Domain
Human Factors
Blindspots in change management, skills, human-AI collaboration, trust, workforce, and culture.
Source
MIT AI Risk Repository — The Ethics of Advanced AI Assistants (Gabriel2024) ↗https://airisk.mit.edu/
Could this happen in your organisation?
A Velinor AI Audit maps your active AI portfolio against the 50+ blindspots and benchmarks against documented sector failures like this one. A board-ready foresight document in 5 weeks.