Why AI explainability is no longer optional
For years, explainability was an afterthought: accuracy was what mattered. That made sense for low-risk applications, but it’s unacceptable when an AI model decides whether someone gets a loan, which medical treatment they’re offered, or whether their job application moves to the next round.
Explainable AI (XAI) is the set of techniques and methods that let people understand, in human terms, how and why an AI model reaches a specific prediction or decision.
With the EU AI Act in force, explainability has gone from good practice to a legal requirement for high-risk AI systems. Systems used in credit, employment, education, social welfare, and healthcare must document how they work, be transparent to the people affected, and allow meaningful human oversight.
Core XAI techniques
SHAP (SHapley Additive exPlanations): based on Shapley values from game theory, SHAP calculates each input variable’s marginal contribution to a specific prediction. It’s the industry standard thanks to its mathematical rigor and its ability to produce both local explanations (why the model predicted this value for this customer) and global ones (which variables matter most to the model overall).
LIME (Local Interpretable Model-agnostic Explanations): fits a simple, interpretable model (linear regression) around each individual prediction to approximate the complex model’s behavior at that point. It’s model-agnostic and very flexible, though less mathematically consistent than SHAP.
Grad-CAM (Gradient-weighted Class Activation Mapping): designed for convolutional neural networks in computer vision. It produces heat maps showing which regions of an image drove the prediction, so you can confirm the model is looking at the right parts of the image (not artifacts or noise).
Attention mechanisms: in Transformer models for NLP, attention weights show which words or tokens the model considered most relevant when generating a given prediction or response.
Inherently interpretable models: instead of adding explainability after the fact, you can choose models that are interpretable by design: logistic regression (coefficients can be read directly), decision trees (the decision path is explicit), or generalized additive models (GAMs), which capture nonlinear relationships while staying interpretable.
Explainability in regulated industries
Banking and finance: Article 22 of the GDPR already established a right to explanation for automated credit decisions. The EU AI Act reinforces it with technical documentation and record-keeping requirements for scoring systems classified as high-risk. Supervisors (the ECB, the EBA, ESMA, and national supervisory authorities) expect institutions to be able to explain how their risk models work.
Insurance: using AI to assess claims or set premiums requires explainability both for the regulator (EIOPA at EU level and national insurance supervisors) and for policyholders, who have the right to know which factors determine their premium.
Healthcare: clinical decision support systems must be explainable to the clinicians who use them. A doctor who doesn’t understand why the system recommends a treatment can’t exercise responsible clinical judgment.
How to add explainability to existing models
Post-hoc explainability can be added to models already in production without retraining them. SHAP, in particular, works independently with virtually any model (gradient boosting, neural networks, random forests). That means companies with models in production can strengthen their regulatory position without throwing away previous work.
The typical process: integrate the SHAP library into the inference pipeline, generate and store SHAP values for every prediction, design explanations for each audience (technical, end user, regulator), and set up a human review process for cases where the explanation points to unexpected factors.
Frequently asked questions
Is there a single explainability standard that regulators accept? There’s no single standard, but SHAP has gained very broad acceptance in financial services thanks to its mathematical properties. The EU AI Act doesn’t prescribe a specific technique; it requires high-risk systems to be transparent enough to allow human oversight. Documentation of the development process, performance metrics by segment, and the ability to explain individual decisions are what regulators look for most in an inspection.
Are explainable models less accurate? There used to be a trade-off between explainability and accuracy: simpler models were easier to interpret but less accurate. Today, gradient boosting algorithms (XGBoost, LightGBM) combined with SHAP deliver high accuracy with robust post-hoc explainability. In many real business problems, the accuracy gap versus deep neural networks is marginal, and the gain in interpretability and regulatory confidence more than justifies the choice.
How do you explain an AI model to a customer whose loan application was declined? The most common practice is to translate the SHAP values of the model’s technical variables into terms the customer understands. For example, instead of saying “the SHAP value of the variable ‘debt_ratio’ is -0.23,” you say “the main factor that weighed against your application is your current level of debt relative to your income.” That translation requires careful design of the communication system, which is part of the XAI implementation work.