Developing LLM-Specific Usability Heuristics: Revealing Critical Gaps in Ethical Design and Safety Protections

Abstract

Large Language Model (LLM) interfaces are increasingly used for high-stakes tasks such as medical consultation, academic work and decision making. However, existing usability frameworks inadequately address LLM-specific ethical and safety concerns. This research thus proposes 15 LLM-specific heuristics with 36 design principles through thematic analysis of 283 design guidelines from Microsoft, Google, IBM, Apple and academic literature. To explore their applicability, nine test prompts were given to ChatGPT and Gemini, and compared with Microsoft’s Human-AI Interaction (HAI) guidelines. The proposed heuristics identified 24 violations across all 9 ChatGPT prompts and 16 violations across 7 of the 9 Gemini prompts, whereas Microsoft’s HAI guidelines identified 5 violations across 3 ChatGPT prompts and none in Gemini prompts. The proposed heuristics also revealed 11 ethical and 12 safety violations beyond Microsoft’s HAI guidelines. The findings suggest that the proposed heuristics and design principles can support developers building LLM interfaces usably, safely and ethically.