Understanding Isn't Enough
Artificial intelligence (AI) is increasingly used to support decisions in areas such as finance, hiring, and risk management. While these systems can be highly accurate, they often operate as “black boxes,” making it difficult to understand how decisions are made. Even more importantly, they usually fail to answer a practical question: how can a decision be changed?
My paper “How do I explain that decision? A new approach combining counterfactuals and sensitivity analyses” introduces a new approach to explainable AI that goes beyond interpretation and supports actionable decision-making. It combines two established tools — counterfactual analysis and Shapley values — to explain not only why an outcome occurred, but also how it could be improved.
The Limit of Traditional Methods
Most existing explanation methods identify which factors influenced a decision, but they do not provide clear guidance on how to achieve a better outcome. For example, a rejected loan applicant might learn that income or savings mattered, but not which change would be most effective in reversing the decision.
The Role of Counterfactual and Shapley Values Analyses
The proposed approach addresses this limitation by focusing on transitions between outcomes. Starting from an individual’s current situation, it identifies a nearby alternative (a “counterfactual”) where the outcome improves—for instance, from not creditworthy to creditworthy. It then quantifies how much each change contributes to the improvement.
To attribute importance to various factors, the method builds on Shapley values, a concept from game theory. However, standard implementation can be misleading in this context, because they are designed to explain individual predictions rather than differences between outcomes. The paper resolves this issue by redefining Shapley values so that they directly explain the variation between two situations.
Realistic and Applicable Explanations
A second innovation addresses a practical limitation of many AI explanations, that they often rely on unrealistic or impossible combinations of features. The proposed method introduces “feasible Shapley values,” which exclude limited cases and restrict the analysis to realistic scenarios.
Benefits for People and Organizations
The methodology offers benefits at multiple levels. For individuals, it identifies which changes are most effective, their direction, and their relative importance. For organizations, applying the analysis across a large number of cases reveals systematic patterns, supports better risk management, and can uncover potential biases or discrimination in the data or models.
Overall, this work reframes explainability in AI, shifting from simply understanding model behavior to supporting better decisions. It provides a practical step forward toward making AI systems not only transparent, but genuinely useful for decision-making.
