mednet.engine.classify.saliency.completeness

Engine and functions for score completeness analysis.

Functions

run(model, datamodule, device_manager, ...)

Evaluate ROAD scores for all samples in a DataModule.

Classes

SigmoidClassifierOutputTarget(category)

Consider output to be a sigmoid.

class mednet.engine.classify.saliency.completeness.SigmoidClassifierOutputTarget(category)[source]

Bases: Module

Consider output to be a sigmoid.

Parameters:

category – The category.

mednet.engine.classify.saliency.completeness.run(model, datamodule, device_manager, saliency_map_algorithm, target_class, positive_only, percentiles, parallel, only_dataset)[source]

Evaluate ROAD scores for all samples in a DataModule.

The ROAD algorithm was first described in [RLB+22]. It estimates explainability (in the completeness sense) of saliency maps by substituting relevant pixels in the input image by a local average, re-running prediction on the altered image, and measuring changes in the output classification score when said perturbations are in place. By substituting the most or least relevant pixels with surrounding averages, the ROAD algorithm estimates the importance of such elements in the produced saliency map. As of 2023, this measurement technique is considered to be one of the state-of-the-art metrics of explainability.

This function returns a dictionary containing most-relevant-first (remove a percentile of the most relevant pixels), least-relevant-first (remove a percentile of the least relevant pixels), and combined ROAD evaluations per sample for a particular saliency mapping algorithm.

Parameters:
  • model (LightningModule) – Neural network model (e.g. pasa).

  • datamodule (LightningDataModule) – The lightning DataModule to iterate on.

  • device_manager (DeviceManager) – An internal device representation, to be used for training and validation. This representation can be converted into a pytorch device or a lightning accelerator setup.

  • saliency_map_algorithm (Literal['ablationcam', 'eigencam', 'eigengradcam', 'fullgrad', 'gradcam', 'gradcamelementwise', 'gradcam++', 'gradcamplusplus', 'hirescam', 'layercam', 'randomcam', 'scorecam', 'xgradcam']) – The algorithm for saliency map estimation to use.

  • target_class (Literal['highest', 'all']) – (Use only with multi-label models) Which class to target for CAM calculation. Can be set to “all” or “highest”. “highest” is default, which means only saliency maps for the class with the highest activation will be generated.

  • positive_only (bool) – If set, saliency maps will only be generated for positive samples (ie. label == 1 in a binary classification task). This option is ignored on a multi-class output model.

  • percentiles (Sequence[int]) – A sequence of percentiles (percent x100) integer values indicating the proportion of pixels to perturb in the original image to calculate both MoRF and LeRF scores.

  • parallel (int) – Use multiprocessing for data processing: if set to -1, disables multiprocessing. Set to 0 to enable as many data processing instances as processing cores available in the system. Set to >= 1 to enable that many multiprocessing instances for data processing.

  • only_dataset (str | None) – If set, will only run this code for the named dataset on the provided datamodule, skipping any other datasets.

Returns:

A dictionary where keys are dataset names in the provide DataModule, and values are lists containing sample information alongside metrics calculated:

  • Sample name

  • Sample target class

  • The model output number used for the ROAD analysis (0, for binary classifers as there is typically only one output).

  • morf: ROAD most-relevant-first average of percentiles 20, 40, 60 and 80 (a.k.a. AOPC-MoRF).

  • lerf: ROAD least-relevant-first average of percentiles 20, 40, 60 and 80 (a.k.a. AOPC-LeRF).

  • combined: Average ROAD combined score by evaluating (lerf-morf)/2 (a.k.a. AOPC-Combined).

Return type:

dict[str, list[Any]]