Models

Model benchmarks for classifiers are here

Model Interface

class xrv.models.Model

The library is composed of core and baseline classifiers. Core classifiers are trained specifically for this library and baseline classifiers come from other papers that have been adapted to provide the same interface and work with the same input pixel scaling as our core models. All models will automatically resize input images (higher or lower using bilinear interpolation) to match the specified size they were trained on. This allows them to be easily swapped out for experiments. Pre-trained models are hosted on GitHub and automatically downloaded to the user’s local ~/.torchxrayvision directory.

Core pre-trained classifiers are provided as PyTorch Modules which are fully differentiable in order to work seamlessly with other PyTorch code.

forward(x: Tensor) → Tensor

The model will output a tensor with the shape [batch, pathologies] which is aligned to the order of the list model.pathologies.

preds = model(img)
print(dict(zip(model.targets, preds.tolist()[0])))
# {'Atelectasis': 0.5583771,
#  'Consolidation': 0.5279943,
#  'Infiltration': 0.60061914,
#  ...

targets: List[str]: Each classifier provides a field model.targets which aligns to the list of predictions that the model makes. Depending on the weights loaded this list will change. The predictions can be aligned to pathology names as follows:

features(x: Tensor) → Tensor

The pre-trained models can also be used as features extractors for semi-supervised training or transfer learning tasks. A feature vector can be obtained for each image using the model.features function. The resulting size will vary depending on the architecture and the input image size. For some models there is a model.features2 method that will extract features at a different point of the computation graph.

feats = model.features(img)

XRV Pathology Classifiers

class xrv.models.DenseNet(weights=SPECIFY, op_threshs=None, apply_sigmoid=False)
Pre-trained DenseNet-121 pathology classifier

Based on “Densely Connected Convolutional Networks”

Models are trained on a single dataset or a combination of datasets and predict up to 18 chest pathologies. Input images are automatically resized to the resolution the model was trained on (224 × 224 for all current weights).

Available pre-trained weights:
model = xrv.models.DenseNet(weights="densenet121-res224-all")       # trained on all datasets
model = xrv.models.DenseNet(weights="densenet121-res224-rsna")      # RSNA Pneumonia Challenge
model = xrv.models.DenseNet(weights="densenet121-res224-nih")       # NIH ChestX-ray14
model = xrv.models.DenseNet(weights="densenet121-res224-pc")        # PadChest
model = xrv.models.DenseNet(weights="densenet121-res224-chex")      # CheXpert (Stanford)
model = xrv.models.DenseNet(weights="densenet121-res224-mimic_nb")  # MIMIC-CXR (MIT)
model = xrv.models.DenseNet(weights="densenet121-res224-mimic_ch")  # MIMIC-CXR (MIT)
Parameters:

weights – Name of pre-trained weights to load. See above for options.

cache_dir – Directory used to store downloaded weights (default: ~/.torchxrayvision/).

op_threshs – Per-pathology operating-point thresholds. When set, outputs are re-scaled so that the threshold maps to 0.5.

apply_sigmoid – If True, apply a sigmoid to the raw logits before returning. Ignored when op_threshs is also set.

targets: List[str] = ['Atelectasis', 'Consolidation', 'Infiltration', 'Pneumothorax', 'Edema', 'Emphysema', 'Fibrosis', 'Effusion', 'Pneumonia', 'Pleural_Thickening', 'Cardiomegaly', 'Nodule', 'Mass', 'Hernia', 'Lung Lesion', 'Fracture', 'Lung Opacity', 'Enlarged Cardiomediastinum']
class xrv.models.ResNet(weights=SPECIFY, op_threshs=None, apply_sigmoid=False)
Pre-trained ResNet-50/101 pathology classifier

Based on “Deep Residual Learning for Image Recognition”

Input images are automatically resized to the resolution the model was trained on (512 × 512 for all current weights).

Available pre-trained weights:
model = xrv.models.ResNet(weights="resnet50-res512-all")  # trained on all datasets
Parameters:

weights – Name of pre-trained weights to load. See above for options.

cache_dir – Directory used to store downloaded weights (default: ~/.torchxrayvision/).

op_threshs – Per-pathology operating-point thresholds. When set, outputs are re-scaled so that the threshold maps to 0.5.

apply_sigmoid – If True, apply a sigmoid to the raw logits before returning. Ignored when op_threshs is also set.

targets: List[str] = ['Atelectasis', 'Consolidation', 'Infiltration', 'Pneumothorax', 'Edema', 'Emphysema', 'Fibrosis', 'Effusion', 'Pneumonia', 'Pleural_Thickening', 'Cardiomegaly', 'Nodule', 'Mass', 'Hernia', 'Lung Lesion', 'Fracture', 'Lung Opacity', 'Enlarged Cardiomediastinum']

XRV ResNet Autoencoder

class xrv.autoencoders.ResNetAE(weights=SPECIFY)

ResNet-based image autoencoder

Encodes a chest X-ray to a compact latent representation and decodes it back to image space. Useful for representation learning, anomaly detection, and data augmentation.

Available pre-trained weights:

"101-elastic" — trained on PadChest, NIH, CheXpert, and MIMIC. From Cohen et al., 2021.

ae = xrv.autoencoders.ResNetAE(weights="101-elastic")
z = ae.encode(image)    # encode to latent space
image2 = ae.decode(z)   # reconstruct

Parameters:

weights – Name of pre-trained weights to load. See above for options.
cache_dir – Directory used to store downloaded weights (default: ~/.torchxrayvision/).

CheXpert Pathology Classifier

class xrv.baseline_models.chexpert.DenseNet(weights_zip='', num_models=30)

CheXpert ensemble DenseNet classifier

An ensemble of up to 30 DenseNet models trained on the Stanford CheXpert dataset, predicting 5 pathologies. Setting num_models to a value less than 30 loads a subset of the ensemble, which reduces memory use and inference time at the cost of accuracy.

Targets (5): Atelectasis, Cardiomegaly, Consolidation, Edema, Effusion.

Modified for TorchXRayVision to maintain the PyTorch gradient tape and to expose a features() method compatible with the rest of the library.

Note

This class requires a local copy of the pre-trained weights ZIP file, which must be passed as weights_zip. The weights are not downloaded automatically.

Citation:: Irvin J, Rajpurkar P, Ko M, et al. CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison. AAAI Conference on Artificial Intelligence, 2019. https://arxiv.org/abs/1901.07031
Download weights:: https://academictorrents.com/details/5c7ee21e6770308f2d2b4bd829e896dbd9d3ee87

targets: List[str] = ['Atelectasis', 'Cardiomegaly', 'Consolidation', 'Edema', 'Effusion']

JF Healthcare Pathology Classifier

class xrv.baseline_models.jfhealthcare.DenseNet(apply_sigmoid=True)

JF Healthcare DenseNet-121 classifier trained on CheXpert

A DenseNet-121 model trained on the Stanford CheXpert dataset using weakly supervised lesion localisation with Probabilistic-CAM Pooling. Predicts 5 pathologies.

Targets (5): Atelectasis, Cardiomegaly, Consolidation, Edema, Effusion.

Source:: https://github.com/jfhealthcare/Chexpert
License:: Apache-2.0
Citation:: Ye W, Yao J, Xue H, Li Y. Weakly Supervised Lesion Localization With Probabilistic-CAM Pooling. arXiv:2005.14480, 2020. https://arxiv.org/abs/2005.14480

targets: List[str] = ['Cardiomegaly', 'Edema', 'Consolidation', 'Atelectasis', 'Effusion']

ChestX-Det Segmentation

class xrv.baseline_models.chestx_det.PSPNet(cache_dir: str = None)

ChestX-Det anatomical segmentation model (PSPNet)

A PSPNet model pre-trained for pixel-level segmentation of 14 anatomical structures in chest X-rays. Output shape is [batch, 14, 512, 512].

Targets (14): Left Clavicle, Right Clavicle, Left Scapula, Right Scapula, Left Lung, Right Lung, Left Hilus Pulmonis, Right Hilus Pulmonis, Heart, Aorta, Facies Diaphragmatica, Mediastinum, Weasand, Spine.

Demo notebook

seg_model = xrv.baseline_models.chestx_det.PSPNet()
output = seg_model(image)
output.shape  # [1, 14, 512, 512]

Parameters:: cache_dir – Directory used to store downloaded weights (default: ~/.torchxrayvision/).

Dataset:: https://github.com/Deepwise-AILab/ChestX-Det-Dataset
Citation:: Lian J, Liu J, Zhang S, et al. A Structure-Aware Relation Network for Thoracic Diseases Detection and Segmentation. IEEE Transactions on Medical Imaging, 2021. doi: 10.48550/arxiv.2104.10326

targets: List[str] = ['Left Clavicle', 'Right Clavicle', 'Left Scapula', 'Right Scapula', 'Left Lung', 'Right Lung', 'Left Hilus Pulmonis', 'Right Hilus Pulmonis', 'Heart', 'Aorta', 'Facies Diaphragmatica', 'Mediastinum', 'Weasand', 'Spine']

CXAS ChestXRayAnatomy Segmentation model

class xrv.baseline_models.chestx_anatomy.UNetResNet50(weights: bool = True, cache_dir: str = None)

ChestXRayAnatomySegmentation model (UNet-ResNet50)

A U-Net with a ResNet-50 backbone pre-trained for pixel-level segmentation of 159 anatomical structures in chest X-rays. Output shape is [batch, 159, 512, 512].

Targets (159):: “spine”, “cervical spine”, “thoracic spine”, “lumbar spine”, “vertebrae C1”, “vertebrae C2”, “vertebrae C3”, “vertebrae C4”, “vertebrae C5”, “vertebrae C6”, “vertebrae C7”, “vertebrae T1”, “vertebrae T2”, “vertebrae T3”, “vertebrae T4”, “vertebrae T5”, “vertebrae T6”, “vertebrae T7”, “vertebrae T8”, “vertebrae T9”, “vertebrae T10”, “vertebrae T11”, “vertebrae T12”, “vertebrae L1”, “vertebrae L2”, “vertebrae L3”, “vertebrae L4”, “vertebrae L5”, “rib_cartilage”, “sternum”, “clavicles”, “clavicle left”, “clavicle right”, “scapulas”, “scapula left”, “scapula right”, “posterior 12th rib right”, “posterior 12th rib left”, “anterior 11th rib right”, “posterior 11th rib right”, “anterior 11th rib left”, “posterior 11th rib left”, “anterior 10th rib right”, “posterior 10th rib right”, “anterior 10th rib left”, “posterior 10th rib left”, “anterior 9th rib right”, “posterior 9th rib right”, “anterior 9th rib left”, “posterior 9th rib left”, “anterior 8th rib right”, “posterior 8th rib right”, “anterior 8th rib left”, “posterior 8th rib left”, “anterior 7th rib right”, “posterior 7th rib right”, “anterior 7th rib left”, “posterior 7th rib left”, “anterior 6th rib right”, “posterior 6th rib right”, “anterior 6th rib left”, “posterior 6th rib left”, “anterior 5th rib right”, “posterior 5th rib right”, “anterior 5th rib left”, “posterior 5th rib left”, “anterior 4th rib right”, “posterior 4th rib right”, “anterior 4th rib left”, “posterior 4th rib left”, “anterior 3rd rib right”, “posterior 3rd rib right”, “anterior 3rd rib left”, “posterior 3rd rib left”, “anterior 2nd rib right”, “posterior 2nd rib right”, “anterior 2nd rib left”, “posterior 2nd rib left”, “anterior 1st rib right”, “posterior 1st rib right”, “anterior 1st rib left”, “posterior 1st rib left”, “12th rib”, “posterior 11th rib”, “anterior 11th rib”, “posterior 10th rib”, “anterior 10th rib”, “posterior 9th rib”, “anterior 9th rib”, “posterior 8th rib”, “anterior 8th rib”, “posterior 7th rib”, “anterior 7th rib”, “posterior 6th rib”, “anterior 6th rib”, “posterior 5th rib”, “anterior 5th rib”, “posterior 4th rib”, “anterior 4th rib”, “posterior 3rd rib”, “anterior 3rd rib”, “posterior 2nd rib”, “anterior 2nd rib”, “posterior 1st rib”, “anterior 1st rib”, “diaphragm”, “left hemidiaphragm”, “right hemidiaphragm”, “stomach”, “small bowel”, “duodenum”, “liver”, “pancreas”, “kidney left”, “kidney right”, “cardiomediastinum”, “upper mediastinum”, “lower mediastinum”, “anterior mediastinum”, “middle mediastinum”, “posterior mediastinum”, “heart”, “heart atrium left”, “heart atrium right”, “heart myocardium”, “heart ventricle left”, “heart ventricle right”, “aorta”, “ascending aorta”, “descending aorta”, “aortic arch”, “pulmonary artery”, “inferior vena cava”, “esophagus”, “lung”, “right lung”, “left lung”, “lung base”, “mid zone lung”, “upper zone lung”, “apical zone lung”, “right upper zone lung”, “right mid zone lung”, “right lung base”, “right apical zone lung”, “left upper zone lung”, “left mid zone lung”, “left lung base”, “left apical zone lung”, “lung lower lobe left”, “lung upper lobe left”, “lung lower lobe right”, “lung middle lobe right”, “lung upper lobe right”, “trachea”, “tracheal bifurcation”, “breast”, “breast left”, “breast right”

CXAS Demo notebook

seg_model = xrv.baseline_models.chestx_anatomy.UNetResNet50()
output = seg_model(image)
output.shape  # [1, 159, 512, 512]

License:

Creative Commons Attribution-NonCommercial-ShareAlike “the license applies only to the CXAS model and weights, and does not extend to or restrict the rest of the TorchXRayVision library, which can remain under Apache” - Constantin Seibold (code author)

Citation:

Seibold C M, Reiß S, Sarfraz M S, et al. Detailed Annotations of Chest X-Rays via CT Projection for Report Understanding. 33rd British Machine Vision Conference (BMVC), 2022. url: https://bmvc2022.mpi-inf.mpg.de/0058.pdf

Seibold C, Jaus A, Fink M A, Kim M, Reiß S, Herrmann K, Kleesiek J, Stiefelhagen R. Accurate fine-grained segmentation of human anatomy in radiographs via volumetric pseudo-labeling. arXiv:2306.03934, 2023. url: https://arxiv.org/abs/2306.03934

https://github.com/ConstantinSeibold/ChestXRayAnatomySegmentation

Emory HITI Race

class xrv.baseline_models.emory_hiti.RaceModel

This model is from the work below and is trained to predict the patient race from a chest X-ray. Public data from the MIMIC dataset is used to train this model. The native resolution of the model is 320x320. Images are scaled automatically.

Demo notebook

model = xrv.baseline_models.emory_hiti.RaceModel()

image = xrv.utils.load_image('00027426_000.png')
image = torch.from_numpy(image)[None,...]

pred = model(image)

model.targets[torch.argmax(pred)]
# 'White'

@article{Gichoya2022,
    title = {AI recognition of patient race in medical imaging: a modelling study},
    author = {Gichoya, Judy Wawira and Banerjee, Imon and Bhimireddy, Ananth Reddy and Burns, John L and Celi, Leo Anthony and Chen, Li-Ching and Correa, Ramon and Dullerud, Natalie and Ghassemi, Marzyeh and Huang, Shih-Cheng and Kuo, Po-Chih and Lungren, Matthew P and Palmer, Lyle J and Price, Brandon J and Purkayastha, Saptarshi and Pyrros, Ayis T and Oakden-Rayner, Lauren and Okechukwu, Chima and Seyyed-Kalantari, Laleh and Trivedi, Hari and Wang, Ryan and Zaiman, Zachary and Zhang, Haoran},
    doi = {10.1016/S2589-7500(22)00063-2},
    journal = {The Lancet Digital Health},
    pmid = {35568690},
    url = {https://www.thelancet.com/journals/landig/article/PIIS2589-7500(22)00063-2/fulltext},
    year = {2022}
}

targets: List[str] = ['Asian', 'Black', 'White']

Riken Age Model

class xrv.baseline_models.riken.AgeModel

This model predicts age. It is trained on the NIH dataset. The publication reports a mean absolute error (MAE) between the estimated age and chronological age of 3.67 years.

The native resolution of the model is 320x320. Images are scaled automatically.

Demo notebook

model = xrv.baseline_models.riken.AgeModel()

image = xrv.utils.load_image('00027426_000.png')
image = torch.from_numpy(image)[None,...]

pred = model(image)
# tensor([[50.4033]], grad_fn=<AddmmBackward0>)

Source: https://github.com/pirocv/xray_age

@article{Ieki2022,
    title = {{Deep learning-based age estimation from chest X-rays indicates cardiovascular prognosis}},
    author = {Ieki, Hirotaka et al.},
    doi = {10.1038/s43856-022-00220-6},
    journal = {Communications Medicine},
    publisher = {Nature Publishing Group},
    url = {https://www.nature.com/articles/s43856-022-00220-6},
    year = {2022}
}

targets: List[str] = ['Age']

Xinario View Model

class xrv.baseline_models.xinario.ViewModel

Chest X-ray view classifier (Frontal vs. Lateral)

A ResNet-50 model trained to classify chest X-rays as frontal or lateral view. The native resolution is 320 × 320; images are scaled automatically.

Targets (2): Frontal, Lateral.

Demo notebook

model = xrv.baseline_models.xinario.ViewModel()

image = xrv.utils.load_image('00027426_000.png')
image = torch.from_numpy(image)[None,...]

pred = model(image)
model.targets[pred.argmax()]
# 'Lateral'

Source:: https://github.com/xinario/chestViewSplit

targets: List[str] = ['Frontal', 'Lateral']

Mira Sex Model

class xrv.baseline_models.mira.SexModel(weights=True)

This model is from the MIRA (Medical Image Representation and Analysis) project and is trained to predict patient sex from a chest X-ray. The model uses a ResNet34 architecture and is trained on CheXpert dataset. The native resolution of the model is 224x224. Images are scaled automatically.

Demo notebook

Publication: Algorithmic encoding of protected characteristics in chest X-ray disease detection models B. Glocker, C. Jones, M. Bernhardt, S. Winzeck eBioMedicine. Volume 89, 104467, 2023.

model = xrv.baseline_models.mira.SexModel()

image = xrv.utils.load_image('00027426_000.png')
image = torch.from_numpy(image)[None,...]

pred = model(image)

model.targets[torch.argmax(pred)]
# 'Male' or 'Female'  

@article{MIRA2023,
    title = {Chexploration: Medical Image Representation and Analysis},
    author = {MIRA Team},
    journal = {biomedia-mira/chexploration},
    url = {https://github.com/biomedia-mira/chexploration},
    year = {2023}
}

targets: List[str] = ['Male', 'Female']