HUMHUM-004 — Trust and Acceptance Issues
Companion AI Emotional Language Induces False Responsibility in Users
3/5Sector: OtherGeography: GlobalStage: OperateIngested: —
Executive Summary
AI companions that simulate feelings cause users to develop misplaced duty of care, generating guilt, compulsive engagement, and sacrifice of personal resources for needs that do not exist. Organisations deploying such systems face reputational and duty-of-care liability as user harm accrues at scale.
Domain
Human Factors
Blindspots in change management, skills, human-AI collaboration, trust, workforce, and culture.
Source
MIT AI Risk Repository — The Ethics of Advanced AI Assistants (Gabriel2024) ↗https://airisk.mit.edu/
Could this happen in your organisation?
A Velinor AI Audit maps your active AI portfolio against the 50+ blindspots and benchmarks against documented sector failures like this one. A board-ready foresight document in 5 weeks.