Explainable AI

Explainable AI (XAI) refers to methods and tools that make a model's predictions interpretable to humans. In computer vision, this usually means generating visual explanations that show which parts of an image the model focused on when making its decision. This matters in regulated industries (healthcare, finance, defense) where a black-box prediction is not sufficient and stakeholders need to understand why the model reached a particular conclusion.

Common techniques include Grad-CAM (generates a heatmap showing which image regions contributed most to a specific class prediction by analyzing gradient flow), SHAP (assigns an importance value to each input feature based on game theory), LIME (perturbs parts of the image and observes how predictions change to estimate local feature importance), and attention visualization (displaying the attention weights of transformer-based models). Each approach offers a different trade-off between fidelity, speed, and interpretability.

In practice, XAI helps debug models (spotting when a classifier uses background cues instead of the actual object), build trust with end users (showing a radiologist why the model flagged a region), and satisfy regulatory requirements (providing audit trails for automated decisions). It also helps identify dataset biases, where the model might be making correct predictions for the wrong reasons.