Models

Model benchmarks for classifiers are here

Model Interface

class xrv.models.Model

The library is composed of core and baseline classifiers. Core classifiers are trained specifically for this library and baseline classifiers come from other papers that have been adapted to provide the same interface and work with the same input pixel scaling as our core models. All models will automatically resize input images (higher or lower using bilinear interpolation) to match the specified size they were trained on. This allows them to be easily swapped out for experiments. Pre-trained models are hosted on GitHub and automatically downloaded to the user’s local ~/.torchxrayvision directory.

Core pre-trained classifiers are provided as PyTorch Modules which are fully differentiable in order to work seamlessly with other PyTorch code.

forward(x: Tensor) → Tensor

The model will output a tensor with the shape [batch, pathologies] which is aligned to the order of the list model.pathologies.

preds = model(img)
print(dict(zip(model.targets, preds.tolist()[0])))
# {'Atelectasis': 0.5583771,
#  'Consolidation': 0.5279943,
#  'Infiltration': 0.60061914,
#  ...
targets: List[str]

Each classifier provides a field model.targets which aligns to the list of predictions that the model makes. Depending on the weights loaded this list will change. The predictions can be aligned to pathology names as follows:

features(x: Tensor) → Tensor

The pre-trained models can also be used as features extractors for semi-supervised training or transfer learning tasks. A feature vector can be obtained for each image using the model.features function. The resulting size will vary depending on the architecture and the input image size. For some models there is a model.features2 method that will extract features at a different point of the computation graph.

feats = model.features(img)

XRV Pathology Classifiers

class xrv.models.DenseNet(weights=SPECIFY, op_threshs=None, apply_sigmoid=False)

Pre-trained DenseNet-121 pathology classifier

Based on “Densely Connected Convolutional Networks”

Models are trained on a single dataset or a combination of datasets and predict up to 18 chest pathologies. Input images are automatically resized to the resolution the model was trained on (224 × 224 for all current weights).

Available pre-trained weights:

model = xrv.models.DenseNet(weights="densenet121-res224-all")       # trained on all datasets
model = xrv.models.DenseNet(weights="densenet121-res224-rsna")      # RSNA Pneumonia Challenge
model = xrv.models.DenseNet(weights="densenet121-res224-nih")       # NIH ChestX-ray14
model = xrv.models.DenseNet(weights="densenet121-res224-pc")        # PadChest
model = xrv.models.DenseNet(weights="densenet121-res224-chex")      # CheXpert (Stanford)
model = xrv.models.DenseNet(weights="densenet121-res224-mimic_nb")  # MIMIC-CXR (MIT)
model = xrv.models.DenseNet(weights="densenet121-res224-mimic_ch")  # MIMIC-CXR (MIT)
Parameters:

  • weights – Name of pre-trained weights to load. See above for options.

  • cache_dir – Directory used to store downloaded weights (default: ~/.torchxrayvision/).

  • op_threshs – Per-pathology operating-point thresholds. When set, outputs are re-scaled so that the threshold maps to 0.5.

  • apply_sigmoid – If True, apply a sigmoid to the raw logits before returning. Ignored when op_threshs is also set.

targets: List[str] = ['Atelectasis', 'Consolidation', 'Infiltration', 'Pneumothorax', 'Edema', 'Emphysema', 'Fibrosis', 'Effusion', 'Pneumonia', 'Pleural_Thickening', 'Cardiomegaly', 'Nodule', 'Mass', 'Hernia', 'Lung Lesion', 'Fracture', 'Lung Opacity', 'Enlarged Cardiomediastinum']
class xrv.models.ResNet(weights=SPECIFY, op_threshs=None, apply_sigmoid=False)

Pre-trained ResNet-50/101 pathology classifier

Based on “Deep Residual Learning for Image Recognition”

Input images are automatically resized to the resolution the model was trained on (512 × 512 for all current weights).

Available pre-trained weights:

model = xrv.models.ResNet(weights="resnet50-res512-all")  # trained on all datasets
Parameters:

  • weights – Name of pre-trained weights to load. See above for options.

  • cache_dir – Directory used to store downloaded weights (default: ~/.torchxrayvision/).

  • op_threshs – Per-pathology operating-point thresholds. When set, outputs are re-scaled so that the threshold maps to 0.5.

  • apply_sigmoid – If True, apply a sigmoid to the raw logits before returning. Ignored when op_threshs is also set.

targets: List[str] = ['Atelectasis', 'Consolidation', 'Infiltration', 'Pneumothorax', 'Edema', 'Emphysema', 'Fibrosis', 'Effusion', 'Pneumonia', 'Pleural_Thickening', 'Cardiomegaly', 'Nodule', 'Mass', 'Hernia', 'Lung Lesion', 'Fracture', 'Lung Opacity', 'Enlarged Cardiomediastinum']

XRV ResNet Autoencoder

class xrv.autoencoders.ResNetAE(weights=SPECIFY)

ResNet-based image autoencoder

Encodes a chest X-ray to a compact latent representation and decodes it back to image space. Useful for representation learning, anomaly detection, and data augmentation.

Available pre-trained weights:

  • "101-elastic" — trained on PadChest, NIH, CheXpert, and MIMIC. From Cohen et al., 2021.

ae = xrv.autoencoders.ResNetAE(weights="101-elastic")
z = ae.encode(image)    # encode to latent space
image2 = ae.decode(z)   # reconstruct
Parameters:

  • weights – Name of pre-trained weights to load. See above for options.

  • cache_dir – Directory used to store downloaded weights (default: ~/.torchxrayvision/).

CheXpert Pathology Classifier

class xrv.baseline_models.chexpert.DenseNet(weights_zip='', num_models=30)

CheXpert ensemble DenseNet classifier

An ensemble of up to 30 DenseNet models trained on the Stanford CheXpert dataset, predicting 5 pathologies. Setting num_models to a value less than 30 loads a subset of the ensemble, which reduces memory use and inference time at the cost of accuracy.

Targets (5): Atelectasis, Cardiomegaly, Consolidation, Edema, Effusion.

Modified for TorchXRayVision to maintain the PyTorch gradient tape and to expose a features() method compatible with the rest of the library.

Note

This class requires a local copy of the pre-trained weights ZIP file, which must be passed as weights_zip. The weights are not downloaded automatically.

Citation:

Irvin J, Rajpurkar P, Ko M, et al. CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison. AAAI Conference on Artificial Intelligence, 2019. https://arxiv.org/abs/1901.07031

Download weights:

https://academictorrents.com/details/5c7ee21e6770308f2d2b4bd829e896dbd9d3ee87

targets: List[str] = ['Atelectasis', 'Cardiomegaly', 'Consolidation', 'Edema', 'Effusion']

JF Healthcare Pathology Classifier

class xrv.baseline_models.jfhealthcare.DenseNet(apply_sigmoid=True)

JF Healthcare DenseNet-121 classifier trained on CheXpert

A DenseNet-121 model trained on the Stanford CheXpert dataset using weakly supervised lesion localisation with Probabilistic-CAM Pooling. Predicts 5 pathologies.

Targets (5): Atelectasis, Cardiomegaly, Consolidation, Edema, Effusion.

Source:

https://github.com/jfhealthcare/Chexpert

License:

Apache-2.0

Citation:

Ye W, Yao J, Xue H, Li Y. Weakly Supervised Lesion Localization With Probabilistic-CAM Pooling. arXiv:2005.14480, 2020. https://arxiv.org/abs/2005.14480

targets: List[str] = ['Cardiomegaly', 'Edema', 'Consolidation', 'Atelectasis', 'Effusion']

ChestX-Det Segmentation

class xrv.baseline_models.chestx_det.PSPNet(cache_dir: str = None)

ChestX-Det anatomical segmentation model (PSPNet)

A PSPNet model pre-trained for pixel-level segmentation of 14 anatomical structures in chest X-rays. Output shape is [batch, 14, 512, 512].

Targets (14): Left Clavicle, Right Clavicle, Left Scapula, Right Scapula, Left Lung, Right Lung, Left Hilus Pulmonis, Right Hilus Pulmonis, Heart, Aorta, Facies Diaphragmatica, Mediastinum, Weasand, Spine.

Demo notebook

seg_model = xrv.baseline_models.chestx_det.PSPNet()
output = seg_model(image)
output.shape  # [1, 14, 512, 512]
_images/segmentation-pspnet.png
Parameters:

cache_dir – Directory used to store downloaded weights (default: ~/.torchxrayvision/).

Dataset:

https://github.com/Deepwise-AILab/ChestX-Det-Dataset

Citation:

Lian J, Liu J, Zhang S, et al. A Structure-Aware Relation Network for Thoracic Diseases Detection and Segmentation. IEEE Transactions on Medical Imaging, 2021. doi: 10.48550/arxiv.2104.10326

targets: List[str] = ['Left Clavicle', 'Right Clavicle', 'Left Scapula', 'Right Scapula', 'Left Lung', 'Right Lung', 'Left Hilus Pulmonis', 'Right Hilus Pulmonis', 'Heart', 'Aorta', 'Facies Diaphragmatica', 'Mediastinum', 'Weasand', 'Spine']

Emory HITI Race

class xrv.baseline_models.emory_hiti.RaceModel

This model is from the work below and is trained to predict the patient race from a chest X-ray. Public data from the MIMIC dataset is used to train this model. The native resolution of the model is 320x320. Images are scaled automatically.

Demo notebook

model = xrv.baseline_models.emory_hiti.RaceModel()

image = xrv.utils.load_image('00027426_000.png')
image = torch.from_numpy(image)[None,...]

pred = model(image)

model.targets[torch.argmax(pred)]
# 'White'

@article{Gichoya2022,
    title = {AI recognition of patient race in medical imaging: a modelling study},
    author = {Gichoya, Judy Wawira and Banerjee, Imon and Bhimireddy, Ananth Reddy and Burns, John L and Celi, Leo Anthony and Chen, Li-Ching and Correa, Ramon and Dullerud, Natalie and Ghassemi, Marzyeh and Huang, Shih-Cheng and Kuo, Po-Chih and Lungren, Matthew P and Palmer, Lyle J and Price, Brandon J and Purkayastha, Saptarshi and Pyrros, Ayis T and Oakden-Rayner, Lauren and Okechukwu, Chima and Seyyed-Kalantari, Laleh and Trivedi, Hari and Wang, Ryan and Zaiman, Zachary and Zhang, Haoran},
    doi = {10.1016/S2589-7500(22)00063-2},
    journal = {The Lancet Digital Health},
    pmid = {35568690},
    url = {https://www.thelancet.com/journals/landig/article/PIIS2589-7500(22)00063-2/fulltext},
    year = {2022}
}
targets: List[str] = ['Asian', 'Black', 'White']

Riken Age Model

class xrv.baseline_models.riken.AgeModel

This model predicts age. It is trained on the NIH dataset. The publication reports a mean absolute error (MAE) between the estimated age and chronological age of 3.67 years.

The native resolution of the model is 320x320. Images are scaled automatically.

Demo notebook

model = xrv.baseline_models.riken.AgeModel()

image = xrv.utils.load_image('00027426_000.png')
image = torch.from_numpy(image)[None,...]

pred = model(image)
# tensor([[50.4033]], grad_fn=<AddmmBackward0>)

Source: https://github.com/pirocv/xray_age

@article{Ieki2022,
    title = {{Deep learning-based age estimation from chest X-rays indicates cardiovascular prognosis}},
    author = {Ieki, Hirotaka et al.},
    doi = {10.1038/s43856-022-00220-6},
    journal = {Communications Medicine},
    publisher = {Nature Publishing Group},
    url = {https://www.nature.com/articles/s43856-022-00220-6},
    year = {2022}
}
targets: List[str] = ['Age']

Xinario View Model

class xrv.baseline_models.xinario.ViewModel

Chest X-ray view classifier (Frontal vs. Lateral)

A ResNet-50 model trained to classify chest X-rays as frontal or lateral view. The native resolution is 320 × 320; images are scaled automatically.

Targets (2): Frontal, Lateral.

Demo notebook

model = xrv.baseline_models.xinario.ViewModel()

image = xrv.utils.load_image('00027426_000.png')
image = torch.from_numpy(image)[None,...]

pred = model(image)
model.targets[pred.argmax()]
# 'Lateral'
Source:

https://github.com/xinario/chestViewSplit

targets: List[str] = ['Frontal', 'Lateral']

Mira Sex Model

class xrv.baseline_models.mira.SexModel(weights=True)

This model is from the MIRA (Medical Image Representation and Analysis) project and is trained to predict patient sex from a chest X-ray. The model uses a ResNet34 architecture and is trained on CheXpert dataset. The native resolution of the model is 224x224. Images are scaled automatically.

Demo notebook

Publication: Algorithmic encoding of protected characteristics in chest X-ray disease detection models B. Glocker, C. Jones, M. Bernhardt, S. Winzeck eBioMedicine. Volume 89, 104467, 2023.

model = xrv.baseline_models.mira.SexModel()

image = xrv.utils.load_image('00027426_000.png')
image = torch.from_numpy(image)[None,...]

pred = model(image)

model.targets[torch.argmax(pred)]
# 'Male' or 'Female'  

@article{MIRA2023,
    title = {Chexploration: Medical Image Representation and Analysis},
    author = {MIRA Team},
    journal = {biomedia-mira/chexploration},
    url = {https://github.com/biomedia-mira/chexploration},
    year = {2023}
}
targets: List[str] = ['Male', 'Female']