Project title: Gifsplanation via Latent Shift: A Simple Autoencoder Approach to Counterfactual Generation for Chest X-rays

Background: The goal of this work is to indicate image features which are relevant to the prediction of a neural network and convey that information to the user by displaying a counterfactual image animation.

The Latent Shift Method: This method works on any pretrained encoder/decoder and classifier which is differentiable. No special considerations are needed during model training. With this approach we want the exact opposite of an adversarial attack but we are using the same idea. We want to perturb the input image so that the classifier reduces its prediction. If we just compute df/dx and move the pixels directly then we will get an imperceivable difference like an adversarial attack. Using a decoder we can regularize the transformation so it will only yield value images.

The encoder takes the input image and encodes it into a latent representation z. Then the decoder reconstructs the image and feeds this image into the classifier. The gradient is computed from the output of the classifier with respect to z. Subtracting the gradient from z and reconstructing the image generates a counterfactual.

Demo: The demonstration below displays the method on 3 different classifiers trained on different datasets. Variation in the concept learned is visible when the explanations are viewed. The same autoencoder is used between the 3 visualizations so the only variable is the classifier. The examples are cherrypicked.

Paper at MIDL 2021, Paper on arXiv

Presentation Video at MIDL 2021

Poster at ICML IMLH 2021

Source code

Colab Demo

View list of true positives and false positives here.

Video speed:

Input Image TorchXRayVision DenseNet121-all
Trained on PadChest, NIH, CheXpert, and MIMIC-CXR datasets
TorchXRayVision DenseNet121-mimic_ch
Trained on the MIMIC-CXR dataset
JF Healthcare DenseNet121
Trained on CheXpert data for the CheXpert challenge
Latent Shift 2D Latent Shift Gif Latent Shift 2D Latent Shift Gif Latent Shift 2D Latent Shift Gif

Prediction of Cardiomegaly

Prediction of Effusion

Prediction of Atelectasis

Prediction of Consolidation

Prediction of Mass

- - - -

Prediction of Pneumothorax

- -

Prediction of Infiltration

- - - -

Prediction of Edema

Prediction of Emphysema

- - - -

Prediction of Fibrosis

- - - -

Prediction of Pneumonia

- -

Prediction of Pleural_Thickening

- - - -

Prediction of Hernia

- - - -

Prediction of Lung Opacity

- -

Joseph Paul Cohen 2021