AI features are easy to announce, but there is often a significant gap between an ambitious vision, “AI magic,” and measurable user value. When designing systems and services in particular, success does not depend on integrating as much AI as possible. It depends on solving the right problem responsibly.
Human-Centered Design, or HCD, provides the foundation for this approach. The international standard ISO [6] defines HCD as an approach to interactive systems development that focuses on users, their needs, and their requirements. It also emphasizes usability knowledge, human factors, and iterative evaluation throughout the product lifecycle.
This becomes especially important for AI systems. Unlike conventional software, AI products may generate non-reducible, inaccurate or intransparent results. Their behavior can also change as models, data, and usage patterns evolve. Research by Yang [14] shows that these characteristics make human-AI interaction particularly difficult to design and test. In the AI Papermaker project, we designed a human-centered AI-based system. In this article we share our lessons learned and elaborate on why AI is not always the answer.

Why an Accurate Model Is Not Enough

Higher model accuracy does not automatically result in better human-AI collaboration. Users also need to understand when they can rely on the system, when they should question its output, and when human intervention is necessary.
Bansal [3] found that strong team performance depends on whether humans and AI can complement each other, rather than on model accuracy alone. Similarly, Okamura and Yamada [9] demonstrate the importance of trust calibration: too much trust can lead to automation bias and poor decisions, while too little trust can result in useful systems being rejected.
Furthermore, principles such as fairness, transparency, accountability, and safety are important from ethical and security perspectives. However, Morley [8] describe a considerable gap between abstract AI ethics principles and their practical implementation. Human-Centered AI must therefore translate these principles into concrete product decisions, responsibilities, tests, and success criteria.

A Practical Process for Businesses

A manageable product-development process can be structured around the Design Council’s [5]: explore the problem, define the focus, develop possible solutions, and deliver and improve the selected approach.

1. Understand the users and their tasks
   If you want to create products that people will adopt, you need to incorporate their demands in the design process. Do not begin with the question, “Where can we use AI?” Begin with the real task people are trying to complete and the challenges they face. Observe workflows, conduct interviews, and investigate users’ goals, frustrations, expectations, skills, and mental models. Consider their level of AI literacy and their existing trust in automated systems. It is also important to identify which activities users want assistance with and which they prefer to perform themselves.
2. Determine whether AI is actually suitable
   Not every problem requires AI. In fact, AI usage bears many risks: costs, ressources and probabilistic outputs [4]. Compare AI-based concepts with simpler alternatives. AI should be selected only when its capabilities match the task, the available data, and the user context. The ultimate goal is not a one-button solution, but rather effective human-AI collaboration [13]. This reflects the human-centered principle described by Shneiderman [12]: AI should enhance human capabilities while preserving meaningful human control, rather than simply replacing people.
3. Translate values into product decisions
   Alongside functional requirements, define values such as control, explainability, privacy, safety, and fairness. Include users, developers, legal specialists, domain experts, and other affected stakeholders. Turn these values into testable hypotheses. Does an explanation help users recognize an unreliable answer? Does a warning reduce inappropriate use? Can users correct the system or override its recommendation? Amershi [2] provide practical guidelines for designing human-AI interaction, including communicating system capabilities, supporting correction, and helping users understand changes in system behavior.
4. Test early and realistically
   Test complete workflows, not only isolated interface elements. Clickable prototypes, simulated AI outputs, or existing AI products can reveal major usability and safety problems before expensive development begins. Evaluation should cover more than user satisfaction and trust. Cognitive aspects like task load and usability-related aspects like ease of use need to be considered. Further, the perceived transparency of an AI system is of importance. Lastly, if you incorporate explainable AI features, these need to be evaluated as well. Refer to Mangold [7] for an overview of evaluation measures and practical guidelines.
5. Connect discovery and development
   Designers, researchers, product managers, and engineers should work together throughout the process. For instance by using frameworks like Dual-Track-Scrum [11]. User needs are central but the incorporation of valuable features requires technical feasability checks. If features cannot be fully implemented they need to be assessed for usefulness. Features with little user value might be postponed instead of partially implemented.
6. Treat launch as the beginning of learning
   It is difficult to simulate real usage conditions during user testing. In the field, users may discover new ways of using the system or learn from frequent interactions with it. Therefore, AI products continue to change after release. To capture future user behaviors, feedback mechanisms, monitoring, escalation paths, and human oversight should be designed from the beginning. The Google People and AI Guidebook [10] recommends creating feedback and control mechanisms that allow users to correct outputs and recover from errors. The NIST risk management framework [1] similarly recommends identifying, measuring, and managing AI risks throughout the system lifecycle.

Beyond Return on Investment

Human-Centered AI is not an optional addition to product strategy. It is a requirement for creating AI products that are useful and responsible at the same time.
Organizations that connect user needs, technical constraints, and potential risks early can reduce costly mistakes while establishing a stronger foundation for responsible and scalable innovation.

References

[1] AI Risk Management Framework | NIST. (2026, June 10). NIST. Retrieved July 23, 2026, from https://www.nist.gov/itl/ai-risk-management-framework

[2] Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., Teevan, J., Kikin-Gil, R., & Horvitz, E. (2019). Guidelines for Human-AI Interaction. CHI ’19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1–13. https://doi.org/10.1145/3290605.3300233

[3] Bansal, G., Nushi, B., Kamar, E., Lasecki, W. S., Weld, D. S., & Horvitz, E. (2019). Beyond accuracy: The role of Mental Models in Human-AI Team performance. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 7, 2–11. https://doi.org/10.1609/hcomp.v7i1.5285

[4] Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the Dangers of Stochastic Parrots. FAccT ’21: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610–623. https://doi.org/10.1145/3442188.3445922

[5] Design Council. (n.d.). The Double Diamond – Design Council. Retrieved July 23, 2026, from https://www.designcouncil.org.uk/resources/the-double-diamond/

[6] ISO 9241-210:2019. (2019). ISO. Retrieved July 23, 2026, from https://www.iso.org/standard/77520.html

[7] Mangold, A., Zietz, J., Weinhold, S., & Pannasch, S. (2025, October 14). On the Design and Evaluation of Human-centered Explainable AI Systems: A Systematic Review and Taxonomy. arXiv.org. https://arxiv.org/abs/2510.12201

[8] Morley, J., Floridi, L., Kinsey, L., & Elhalal, A. (2019). From What to How: An Initial Review of Publicly Available AI Ethics Tools, Methods and Research to Translate Principles into Practices. Science and Engineering Ethics, 26(4), 2141–2168. https://doi.org/10.1007/s11948-019-00165-5

[9] Okamura, K., & Yamada, S. (2020). Adaptive trust calibration for human-AI collaboration. PLoS ONE, 15(2), e0229132. https://doi.org/10.1371/journal.pone.0229132

[10] People + AI Guidebook. (n.d.). Retrieved July 23, 2026, from https://pair.withgoogle.com/guidebook/

[11] Salimi, S. (n.d.). Dual-Track scrum. Agile Academy. Retrieved July 23, 2026, from https://www.agile-academy.com/de/product-owner/dual-track-scrum/

[12] Shneiderman, B. (2020). Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy. International Journal of Human-Computer Interaction, 36(6), 495–504. https://doi.org/10.1080/10447318.2020.1741118

[13] Wang, D., Churchill, E., Maes, P., Fan, X., Shneiderman, B., Shi, Y., & Wang, Q. (2020). From Human-Human Collaboration to Human-AI Collaboration. CHI EA ’20: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, 1–6. https://doi.org/10.1145/3334480.3381069

[14] Yang, Q., Steinfeld, A., Rosé, C., & Zimmerman, J. (2020). Re-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design. CHI ’20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–13. https://doi.org/10.1145/3313831.3376301