HUMHUM-004 — Trust and Acceptance Issues
Overtrust in AI Financial Assistants Leads to Unchallenged Harmful Recommendations
3/5Sector: FinanceGeography: GlobalStage: OperateIngested: —
Executive Summary
Users systematically misjudge AI assistant competence in finance, accepting flawed or harmful recommendations without scrutiny due to inflated capability claims and the persuasive fluency of conversational systems. Boards face material conduct risk and regulatory exposure where AI tools operate beyond validated competence thresholds without adequate human oversight controls.
Domain
Human Factors
Blindspots in change management, skills, human-AI collaboration, trust, workforce, and culture.
Source
MIT AI Risk Repository — The Ethics of Advanced AI Assistants (Gabriel2024) ↗https://airisk.mit.edu/
Could this happen in your organisation?
A Velinor AI Audit maps your active AI portfolio against the 50+ blindspots and benchmarks against documented sector failures like this one. A board-ready foresight document in 5 weeks.