Explainable AI

Minute Read

What is Explainable Artificial Intelligence (XAI)?

Explainable Artificial Intelligence (XAI) is a recent transformative approach in the field of artificial intelligence, focusing on enhancing the transparency and understandability of Artificial Intelligence (AI) systems for humans [1]. The term “XAI” was formulated in the year 2019 by the Defense Advanced Research Projects Agency (DARPA), which is an US-American authority of the ministry of defense [2]. This booming surge in research and development within XAI is driven by a growing recognition of its critical importance, contrasting with traditional “black box” AI models.

XAI aims to demystify AI decision-making processes by developing techniques and algorithms that provide insights into how and why AI systems reach specific outcomes [3]. By allowing us to peer into the “mind” of AI, XAI aims to foster trust, more effective utilization, and alignment of AI actions with our expectations and values. XAI could therefore effectively bridge the gap between advanced AI technology and human comprehension in the pursuit of responsible and accountable AI applications.

Figure 1: XAI as a solution to the black box problem.

Why XAI Matters – Solving the Mystery of Black Box AI

Ever wondered how AI systems make decisions that sometimes feel like magic? Often, they operate like impenetrable black boxes, leaving users and stakeholders in the dark about the reasoning behind their choices [3]. This opacity is particularly evident in fields like healthcare, where AI-driven diagnoses can have life-altering consequences. In this context, XAI could help professionals by clarifying AI-decision-making, enhance trust and enhancing transparency, trust, and accountability between human users and complex AI models [4]. For example, AI is currently used in diagnostic imaging, helping professionals to detect anomalous patterns in X-rays [5]. In an era where AI plays an increasingly pivotal role in our lives, XAI is not just a technical advancement but a fundamental driver of responsible and ethical AI adoption, ensuring that AI technology aligns with human values and expectations.

The below graphic shows the difference between white box models which are inherently interpretable and black box models, which need XAI techniques to be explainable. This framework was developed by us, adapting existing definitions of common XAI terminology.

An image comparing BlackBox & WhiteBox methods. BlackBox models are, in contrast to WhiteBox-models not interpretable by design. Explainability needs to be added to these models to make them transparent. Also terminology is explained (Explainabilty, XAI, XUI, Interpretability, Comprehensibility). Please take a look into our glossary for definitions. — Figure 2: From black box and white box models to comprehensible models. This chart was created by our team based on common XAI terminology in the literature.

What are the Benefits of XAI?

XAI is like a clear window into the AI decision-making process and plays a pivotal role in making the complex world of AI more understandable to humans. At its core, XAI achieves this by offering transparency into the decision-making processes of AI algorithms. It does so through various techniques, including model visualization, feature importance analysis, and generating explanations for AI predictions or recommendations. [1] By employing these methods, XAI sheds light on why AI systems make specific decisions, providing users with insights they can comprehend. This enhanced transparency also benefits non-technical members of organizational teams, bridging the gap between them and the data scientists.

For developers, XAI can serve as a valuable tool for debugging AI models. It provides a deeper understanding of the decision-making process, enabling more effective troubleshooting [6].

What are the Stages of XAI?

In general, explainability can be applied throughout the entire AI development pipeline: before, during and after modeling [7].

In the pre-modeling phase of XAI, data is cleansed, preprocessed, and relevant features are selected to establish the groundwork for transparent and reliable AI models, mitigating biases and enhancing data quality.

In the explainable modeling phase of XAI, the goal is to create AI models that provide accurate predictions while being transparent and interpretable. This is achieved through techniques like using interpretable models, feature engineering, and model-independent explainability methods, enabling humans to understand and scrutinize the AI model's decision-making process.

In the post-modeling stage of XAI, the emphasis shifts to the evaluation and validation of the AI model's transparency and interpretability. This phase involves assessing the model's performance, refining explanations, and ensuring that the AI system continues to meet transparency and fairness standards as it operates in real-world scenarios.

What are some XAI Techniques?

XAI encompasses a range of techniques and methods aimed at making AI systems more transparent and interpretable.

Local vs. Global Explanations

In the realm of XAI, explanations are often divided into global explanations and local explanations [8]. Global explanations tackle the overarching question of "how does a model work?" They offer a broad view of the model's decision-making process, often presented through visual charts, mathematical formulae, or model graphs. These explanations aim to provide a holistic, top-down understanding of the model's behavior. On the other hand, local explanations focus on the "why" behind specific predictions for individual inputs. They are more granular, attributing predictions to certain features of the data or aspects of the model's algorithm. While global explanations give us a general framework of the model, local explanations delve into the specifics of individual decisions, offering a bottom-up perspective.

Counterfactual Explanations

Counterfactual explanations provide alternative input scenarios that, if applied, would change the model's prediction. They aim to help users understand how to achieve desired outcomes [9].

Imagine you applied for a loan, but it was denied by an AI-powered lending system. A counterfactual explanation in this scenario would provide you with an alternative set of actions or conditions that, if met, would have led to your loan being approved. For instance, the explanation might suggest that if your credit score had been 20 points higher, your loan application would have been accepted, giving you a clear actionable insight into improving your chances in the future.

Saliency Maps

Saliency maps can be particularly useful for understanding and interpreting machine learning algorithms. A Saliency Map presents an image highlighting the importance of a pixel i.e. by varying brightness [10].

LIME (Local Interpretable Model-Agnostic Explanations)

LIME is a widely used tool for generating local explanations for machine learning models. Local means that it helps users understand why a specific model made a particular prediction for a given instance. LIME explains machine learning model predictions by perturbing the input data and observing the resulting changes in predictions. Then, it uses these observations to create a simple model around the prediction to highlight which features most influenced the model's decision [11].

SHAP (SHapley Additive exPlanations)

SHAP is a versatile framework that provides unified explanations for a wide range of machine learning models. It assigns Shapley values to features, based on cooperative game theory and offers a coherent approach to explaining model predictions. Like a player in a game, each feature contributes to the model’s prediction. Just as Shapley values fairly distribute the payoff among players in a game, SHAP assigns a fair value to each feature based on its contribution to the prediction outcome [12].

Suppose you're trying to understand why a specific movie recommendation was made by a streaming platform's AI algorithm. SHAP can help by breaking down the contribution of each feature, like genre preferences and viewing history, to the recommendation. For example, SHAP might reveal that the recommendation was primarily influenced by your recent interest in science fiction films, making the suggestion more transparent and insightful.

Figure 6: Prediction of a movie recommendation explained by SHAP. This chart was created for illustrative purposes.

Challenges and Limitations - The Realities of XAI

XAI aims to make complex AI models more transparent and understandable, but it faces several challenges:

Balancing Performance with Transparency: A key challenge in XAI is balancing model performance with transparency. XAI often involves simplifying complex AI models for better understandability, potentially reducing predictive accuracy. This trade-off between maintaining high performance and ensuring interpretability is significant [13].
Security Concerns and Adversarial Attacks: XAI's focus on making AI decisions clear can open new vulnerabilities, especially to adversarial attacks. These attacks use misleading data to manipulate the model's output, exploiting XAI's transparency. This raises serious concerns about the reliability and robustness of AI systems and questions their security and trustworthiness [14].
Computational Intensiveness: Traditional XAI methods often suffer from high computational complexity, which can hinder their deployment in real-time systems. This computational demand limits the practicality of XAI, especially in scenarios requiring rapid decision-making. Overcoming this challenge involves balancing the trade-off between the efficiency and fidelity of explanations [15].
Ethical Considerations (EU AI act): The proposed EU AI act plays a pivotal role in regulating AI. This act concerns AI across various sectors and emphasizes the categorization of AI applications regarding their risk potential. It introduces strict norms against harmful AI practices and requires rigorous assessments for high-risk systems. This regulation aligns with the global push towards ethical AI, mandating organizations to ensure their AI models comply with stringent data privacy and ethical standards [16]. As AI systems are evolving rapidly, this act is crucial to ensure a responsible and transparent AI utilization.

Learn More About XAI

If you're interested in delving deeper into XAI, here are some resources to explore further. These articles, research papers, and tools can help you expand your knowledge of this exciting field.

AI Paper Maker's Role in XAI

The field of XAI is evolving rapidly with researchers working to make AI even more transparent and trustworthy. With our project, “AI Paper Maker”, we're committed to advancing XAI. Our primary objective is to delve into the development of an XAI architecture capable of processing data in a transparent manner, prioritizing accuracy as a guiding principle. To achieve this goal, we want to combine promising XAI techniques and validate their effectiveness with potential users.

Through this research, we aim to lay the groundwork for creating a prototype application that will simplify and enhance the scientific paper-writing process.

AI PAPER MAKER