🧠 My Model Could Predict. But Could I Understand Why?
In my previous posts, I focused on building and improving my first CNN for brain tumor classification using MRI images.
The model could take an MRI image and predict one of four classes:
- Glioma
- Meningioma
- Pituitary
- No Tumor
But after getting the prediction, I started asking a different question:
Why did the model make this prediction?
A classification result such as:
Prediction: Glioma
Confidence: 91%
tells me what the model predicted.
It doesn't tell me which part of the MRI influenced that prediction.
That is where Grad-CAM became useful in my project.
🤔 Why Is Explainability Important?
For a normal image classification problem, seeing the predicted class might be enough for a basic demonstration.
But medical imaging is different.
If a model predicts that an MRI contains a tumor, I don't want to look only at the final class.
I also want to investigate:
- Where was the model focusing?
- Was it looking at a meaningful region?
- Was it focusing on the suspected tumor area?
- Could the model be using some irrelevant visual pattern?
This doesn't mean that an explanation automatically makes the model correct.
Instead, it gives us another way to inspect the model's behavior.
That became an important part of my project.
🔍 What Is Grad-CAM?
Grad-CAM (Gradient-weighted Class Activation Mapping) is a technique used to visualize the regions of an image that contributed to a CNN's prediction for a particular class.
Instead of only getting:
Input MRI → CNN → Prediction
we can visualize something closer to:
Input MRI
↓
CNN prediction
↓
Grad-CAM
↓
Important regions highlighted
The result is usually represented as a heatmap.
Regions with stronger activation are highlighted more strongly, allowing us to visually inspect where the model was focusing.
⚙️ How Grad-CAM Works — Conceptually
The mathematics behind Grad-CAM can become fairly detailed, but the basic idea is easier to understand as a sequence.
Step 1 — Give the MRI image to the CNN
The preprocessed MRI image is passed through the trained CNN.
MRI Image
↓
Preprocessing
↓
CNN
Step 2 — Get the model's prediction
The CNN produces probabilities for the different classes.
For example:
Glioma → 0.91
Meningioma → 0.04
Pituitary → 0.03
No Tumor → 0.02
The highest probability becomes the predicted class.
Step 3 — Select the target class
Grad-CAM focuses on the class we want to investigate.
For example:
Target class → Glioma
We then examine how the model's internal feature maps contributed to that class.
Step 4 — Calculate gradients
Grad-CAM uses the gradients of the target class with respect to the feature maps of a convolutional layer.
Conceptually:
Target class
↓
Gradients
↓
Feature maps
The gradients indicate how important different feature-map activations are for the selected prediction.
Step 5 — Create the heatmap
The feature maps are combined using their importance weights to produce a coarse spatial heatmap.
Feature Maps
+
Gradient Weights
↓
Grad-CAM Heatmap
Step 6 — Overlay the heatmap
Finally, the heatmap can be resized and overlaid on the original MRI.
This gives us a visual representation of the regions that were important to the model's prediction.
🧩 The Complete Grad-CAM Pipeline
Here is the complete process I used conceptually in my project:
The important thing to remember is that Grad-CAM comes after the model has already learned and made a prediction.
It doesn't change the prediction.
It helps us investigate it.
🧠 Grad-CAM in My Brain MRI Project
I applied Grad-CAM to my trained CNN to visualize the regions that contributed to its predictions.
The basic workflow was:
MRI Image
↓
Preprocessing
↓
Trained CNN
↓
Predicted Class
↓
Gradients of Target Class
↓
Feature Maps
↓
Grad-CAM Heatmap
↓
Overlay on MRI
This allowed me to move from simply looking at the prediction to actually inspecting the model's focus.
🔬 Grad-CAM in Action
The following example shows the idea more clearly.
The original MRI is passed through the model.
The model produces a prediction, and Grad-CAM generates a heatmap showing the regions associated with that prediction.
The final overlay makes the explanation easier to interpret visually.
One important detail:
The highlighted region represents where the model was focusing for that prediction. It should not be interpreted as an exact tumor boundary.
This distinction is especially important in medical imaging.
🖼️ Example 1 — No-Tumor Prediction
Here is an example where the model predicted a no-tumor image.
The heatmap provides a visual indication of the regions that contributed to the prediction.
But it is important not to assume that every highlighted pixel has a direct medical meaning.
Grad-CAM is showing the model's learned visual associations—not a medically validated segmentation mask.
🧠 Example 2 — Tumor Prediction
Now consider an MRI containing a visible tumor region.
In this example, the strongest activation is around the visible abnormal region.
That is interesting because it suggests that the model was focusing on a region that appears relevant to the classification.
However, this should still be treated as model interpretation, not proof that the model has correctly identified the tumor.
That distinction matters.
💡 Why Explainability Matters
This changed how I looked at my CNN project.
Before using Grad-CAM, I mainly focused on:
Accuracy
Precision
Recall
F1-score
Prediction
After exploring explainability, I also started asking:
What is the model looking at?
Why might it be making this prediction?
Is the model focusing on a meaningful region?
Grad-CAM doesn't make a model automatically trustworthy.
Instead, it gives us another tool for examining model behavior and identifying potential problems.
⚠️ What Grad-CAM Can and Cannot Tell Us
This was probably the most important thing I learned.
✅ Grad-CAM can help us:
- Visualize important regions for a prediction
- Inspect what the CNN appears to focus on
- Identify potentially suspicious model behavior
- Compare model attention across different images
- Make CNN predictions easier to investigate
❌ Grad-CAM cannot guarantee:
- That the prediction is correct
- That the highlighted region is the exact tumor boundary
- That the model understands the medical meaning of the highlighted area
- That the model is clinically reliable
- That the explanation itself proves causation
In other words:
A heatmap is an explanation of model behavior, not a medical diagnosis.
🧪 What I Learned From Implementing Grad-CAM
Implementing Grad-CAM taught me something beyond the technique itself.
Earlier, I thought machine learning was mainly about improving metrics.
Now I see another important side:
Understanding why a model behaves the way it does is also part of building a good ML system.
The process also made me more careful about how I interpret model results.
A high-confidence prediction can still be wrong.
And a visually convincing heatmap does not automatically mean the model has learned the correct reasoning.
🧠 From Building a CNN to Understanding It
This four-part learning journey changed how I approached my first CNN project.
Part 1 — Building my first CNN
I learned what changes when CNN theory becomes actual code.
Part 2 — Mistakes
I learned that good-looking accuracy doesn't necessarily mean good model performance.
Part 3 — Building the model
I understood the basic workflow from preprocessing to training and evaluation.
Part 4 — Grad-CAM
I learned to look beyond the prediction and investigate where the model was focusing.
The bigger lesson for me was:
Building the model is only one part of machine learning. Understanding its behavior is another.
🚀 What's Next?
This project started as an academic exercise to understand CNN-based image classification.
It gradually became an opportunity to explore:
- Data preprocessing
- CNN architecture
- Model evaluation
- Multi-class classification
- Explainable AI
- Responsible interpretation of AI predictions
There is still a lot I want to improve, especially around model evaluation, dataset quality, explainability techniques, and making the overall system more robust.
🔗 Explore the Project
The complete implementation is available on GitHub:
I also deployed a working demonstration using Streamlit:
The project includes the CNN-based classification workflow along with Grad-CAM-based visualization for model interpretation.
Important: This project is an academic decision-support prototype for learning and experimentation. It is not intended to replace professional medical diagnosis or clinical decision-making.
💬 Final Thought
My biggest takeaway from this project wasn't simply:
"I built a CNN."
It was:
"I started learning how to question what my model was doing."
That shift—from getting a prediction to investigating the prediction—was one of the most valuable things I learned while building this project.
👨🏻💻 Let's Connect
I'm continuing to learn and build projects around Java, web development, machine learning, and AI/ML.
If you're also learning CNNs, computer vision, or explainable AI, I'd be happy to hear about your experience.
What was the first thing you used to understand what your ML model was actually learning?
Let's connect and learn together. 🚀
👨🏻💻 About Me
I'm Tanmay P. Tawade, a Full Stack Developer focused on Java and web development, with an interest in AI/ML and practical software projects.
I write about what I learn while building projects, experimenting with technologies, and turning concepts into working software.
