Explainable AI

Within artificial intelligence (AI), explainable AI (XAI), generally overlapping with interpretable AI, interpretable machine learning and explainable machine learning (XML), is a field of research that explores methods that provide humans with the ability to exercise intellectual oversight over AI algorithms.[1][2] The main focus is on the reasoning behind the decisions or predictions made by the AI algorithms,[3] to make them more understandable and transparent.[4] This addresses the need for users to know the AI-safety of the algorithms and scrutinize the automated decision-making in applications.[5][6] XAI counters the "black box" tendency of machine learning, where even the AI's designers cannot explain why it arrived at a specific decision.[7][8] https://en.wikipedia.org/wiki/Explainable_artificial_intelligence

• Some scholars have suggested that explainability in AI should be considered a goal secondary to AI effectiveness, and that encouraging the exclusive development of XAI may limit the functionality of AI more broadly.[105][106] Critiques of XAI rely on developed concepts of mechanistic and empiric reasoning from evidence-based medicine to suggest that AI technologies can be clinically validated even when their function cannot be understood by their operators.[105]
  • Some researchers advocate the use of inherently interpretable machine learning models, rather than using post-hoc explanations in which a second model is created to explain the first. This is partly because post-hoc models increase the complexity in a decision pathway and partly because it is often unclear how faithfully a post-hoc explanation can mimic the computations of an entirely separate model.[23] However, another view is that what is important is that the explanation accomplishes the given task at hand, and whether it is pre or post-hoc does not matter. If a post-hoc explanation method helps a doctor diagnose cancer better, it is of secondary importance whether it is a correct/incorrect explanation.
• see AI Alignment