The Central Challenge in Explainable AI: Channel Capacity

Posted on

7 minute read

Explainable AI is about communication: we want to tell people how or why a machine learning model is making certain decisions. Why is this so difficult? In this post I take an information-theoretic perspective: the amount of information that people can absorb is fundamentally limited, so the central challenge in explainable AI is to say something useful within a limited number of bits. In other words, the communication channel between the machine learning model and the person has limited capacity.

Explanations as Messages over a Communication Channel

I have made a cartoon illustration in Figure 1, which shows how we can see the task of explainability as being analogous to setting up a communication channel in information theory: the object to communicate is (some aspect of) a complicated function \(f\) that is learned by a neural network or other machine learning model; then explanations encode this object, and they are transmitted to a user. But users can only absorb a limited amount of information, so there is a bandwidth limit on how much information explanations can maximally contain. We may perhaps stretch this limit slightly by designing good user interfaces, but the central challenge is to restrict what we want to communicate, so that explanations can be sufficiently concise. As a consequence, we will never be able to communicate millions of internal parameters from e.g. a neural network in their raw form.

Figure 1: The central problem of explainability: no matter which user interface we design, explanations can only communicate a limited amount of information to users. For instance, we cannot communicate to a user all the millions of parameters of a large neural network.

Limiting the Amount of Information

The conclusion from the communication channel perspective is that we must limit the amount of information that we want to communicate. So what are possible ways do this?

Option 1: Restrict Attention to Simple Models

The most attractive solution, whenever possible, is to start out with a model \(f\) that is sufficiently simple. For instance, if \(f\) can be represented by a small decision tree, then we can describe the structure of the tree in a limited number of bits, so we can communicate \(f\) in full.

This is the approach advocated by Rudin and collaborators, who have put much effort into identifying high-stakes machine learning applications where interpretable models like small decision trees can provide high accuracy (missing reference). Even though such models are very restricted (by definition), there still turn out to be surprisingly many such applications [1]. Relatedly, Bressan et al. [2] have recently developed a mathematically sophisticated characterization that captures how well models \(f\) can be approximated by decision trees.

Option 2: Select Information that is Relevant to a Specific Task

Can we still do something when \(f\) is inherently complicated? Yes, but we will have to choose which aspects of \(f\) we are going to communicate. How do we select the right aspects? This may be possible if we know that the user is trying to accomplish a specific task, because then we can communicate only the specific aspects of \(f\) that are relevant to that particular task.

For instance, maybe the user only cares about verifying that people similar to them get approximately the same predictions. In this case we may communicate the range of values taken by \(f\) in a neighborhood of the user’s features, instead of communicating all of \(f\). Another example comes from algorithmic recourse [3]: if a user’s bank loan is denied by a credit scoring algorithm \(f\), they may want to know how to improve their loan application to get the loan when they reapply. In this case, it may be sufficient to communicate a so-called counterfactual option to the user that tells them the easiest changes they can make to their application to get a loan.1

Option 3: Exploit Common Background Knowledge

Finally, a third option to consider is that the user may have domain knowledge that is relevant to understanding \(f\). In this case it is possible that \(f\) is very complicated by itself, but, conditional on the user’s background knowledge, \(f\) can be described with a limited number of bits. This fits, for instance, with concept bottleneck models [6], which train \(f\) to be a simple function \(g\) of a given set of concepts that the user already understands.2 In this case, we do not need to communicate the concepts, because the user already knows them, and it is sufficient to communicate \(g\), which is feasible.

Conclusion

In this post, I have described a general perspective on explainability that gives a way to think about the fundamental limitations of what can be explained. An important issue I haven’t touched upon, is that it is not necessarily sufficient to transmit the desired information over a communication channel, because the receiving party (the user) also needs to be able to decode the message. For explainability, this means that certain users will have limitations on the types of messages they can understand, which may force the explanations to use a less efficient coding scheme than would be information-theoretically possible.

References

  1. L. Semenova, C. Rudin, and R. Parr, “On the Existence of Simpler Machine Learning Models,” in Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, New York, NY, USA, 2022, pp. 1827–1858, doi: 10.1145/3531146.3533232.
  2. M. Bressan, N. Cesa-Bianchi, E. Esposito, Y. Mansour, S. Moran, and M. Thiessen, “A Theory of Interpretable Approximations,” in Proceedings of Thirty Seventh Conference on Learning Theory (COLT), 2024, vol. 247, pp. 648–668, [Online]. Available at: https://proceedings.mlr.press/v247/bressan24a.html.
  3. A.-H. Karimi, G. Barthe, B. Schölkopf, and I. Valera, “A survey of algorithmic recourse: definitions, formulations, solutions, and prospects,” arXiv preprint arXiv:2010.04050, 2021.
  4. H. Fokkema, D. Garreau, and T. van Erven, “The Risks of Recourse in Binary Classification,” in Proceedings of the 27th International Conference on Artificial Intelligence and Statistics (AISTATS), 2024, vol. 238, pp. 550–558, [Online]. Available at: https://proceedings.mlr.press/v238/fokkema24a.html.
  5. S. Milli, J. Miller, A. D. Dragan, and M. Hardt, “The Social Cost of Strategic Classification,” 2019.
  6. P. W. Koh et al., “Concept Bottleneck Models,” in Proceedings of the 37th International Conference on Machine Learning, 2020, vol. 119, pp. 5338–5348, [Online]. Available at: https://proceedings.mlr.press/v119/koh20a.html.
  7. Y. Goyal, A. Feder, U. Shalit, and B. Kim, “Explaining classifiers with causal concept effect (CaCE),” arXiv preprint arXiv:1907.07165, 2019.
  8. A. Margeloiu, M. Ashman, U. Bhatt, Y. Chen, M. Jamnik, and A. Weller, “Do Concept Bottleneck Models Learn as Intended?,” arXiv preprint arXiv:2106.13314, 2021.
  9. A. Mahinpei, J. Clark, I. Lage, F. Doshi-Velez, and W. Pan, “Promises and Pitfalls of Black-Box Concept Learning Models,” arXiv preprint arXiv:2106.13314, 2021.
  10. M. E. Zarlenga et al., “Towards robust metrics for concept representation evaluation,” 2023.
  11. H. Fokkema, T. van Erven, and S. Magliacane, “Sample-efficient Learning of Concepts with Theoretical Guarantees: from Data to Concepts without Interventions,” in Advances in Neural Information Processing Systems (NeurIPS), 2025, vol. 38, pp. 111783–111843, [Online]. Available at: https://proceedings.neurips.cc/paper_files/paper/2025/file/a23fa41edb52c314c058fd5ce97217d5-Paper-Conference.pdf.

  1. I was initially very excited about algorithmic recourse because it sets a clear goal that the user is trying to achieve. But I have become more skeptical after realizing that providing algorithmic recourse at the population level will always undermine the accuracy of \(f\), as we show in [4]. For instance, in the case of bank loans, this leads to many more customers getting a loan they are unable to repay, which is bad not just for the bank, but also for the customers that recourse is intended to help. It is therefore no longer clear to me that the goal of algorithmic recourse is actually desirable. Indeed, in the literature on strategic classification, the same bank loan example is used to motivate that banks should defend against customers who are trying to game the system [5]. ↩

  2. Learning concepts that really mean what they appear to mean is surprisingly tricky. For instance, spurious correlations in the training data may lead to learned concepts that do not capture the intended semantics [7], [8] or encode more information than intended [9], [10]. We have recently contributed to a line of work that uses ideas from causal representation learning to try to address these issues [11]. ↩

Other posts you might like: