Command-line Interface

This section contains an overview of command-line applications shipped with this package.

mednet

Medical image analysis AI toolbox.

mednet [OPTIONS] COMMAND [ARGS]...

classify

Image classification specialized commands.

mednet classify [OPTIONS] COMMAND [ARGS]...

evaluate

Evaluate predictions (from a model) on a classification task.

It is possible to pass one or several Python files (or names of mednet.config entry points or module names) as CONFIG arguments to the command line which contain the parameters listed below as Python variables. The options through the command-line (see below) will override the values of configuration files. You can run this command with <COMMAND> -H example_config.py to create a template config file.

mednet classify evaluate [OPTIONS] [CONFIG]...

Options

-p, --predictions <predictions>

Required Filename in which predictions are currently stored

-o, --output-folder <output_folder>

Required Directory in which to store results (created if does not exist)

Default:

'results'

-t, --threshold <threshold>

Required This value is used to define positives and negatives from probability outputs in predictions, and report performance measures on binary classification tasks. It should either come from the training set or a separate validation set to avoid biasing the analysis. Optionally, if you provide a multi-split set of predictions as input, this may also be the name of an existing split (e.g. validation) from which the threshold will be estimated (by calculating the threshold leading to the highest F1-score on that set) and then applied to the subsequent sets. This value is not used for multi-class classification tasks.

Default:

0.5

-b, --binning <binning>

Required The binning algorithm to use for computing the bin widths and distribution for histograms. Choose from algorithms supported by numpy.histogram(), or a simple integer indicating the number of bins to have in the interval [0, 1].

Default:

'50'

-P, --plot, --no-plot

Required If set, then also produces figures containing the plots of performance curves and score histograms.

Default:

True

-I, --credible-regions, --no-credible-regions

Required If set, then evaluates also credible intervals. Notice the evaluation of ROC-AUC and Average Precision confidence margins can be rather slow for larger datasets.

Default:

False

-s, --seed <seed>

Seed to use for the random number generator (used when doing Monte Carlo ” simulations required for the evaluation of credible regions for F1-score).

Default:

42

-v, --verbose

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

-H, --dump-config <dump_config>

Name of the config file to be generated

Arguments

CONFIG

Optional argument(s)

Examples:

  1. Run evaluation on an existing prediction output:

    mednet classify evaluate -vv --predictions=path/to/predictions.json
    
  2. Run evaluation on an existing prediction output, tune threshold a priori on the validation set:

    mednet classify evaluate -vv --predictions=path/to/predictions.json --threshold=validation
    

saliency

Generate, evaluate and view saliency maps.

mednet classify saliency [OPTIONS] COMMAND [ARGS]...
completeness
Evaluate saliency map algorithm completeness using RemOve And Debias

(ROAD).

For the selected saliency map algorithm, evaluates the completeness of explanations using the RemOve And Debias (ROAD) algorithm. The ROAD algorithm was first described at [RLB+22]. It estimates explainability (in the completeness sense) of saliency mapping algorithms by substituting relevant pixels in the input image by a local average, re-running prediction on the altered image, and measuring changes in the output classification score when said perturbations are in place. By substituting most or least relevant pixels with surrounding averages, the ROAD algorithm estimates the importance of such elements in the produced saliency map. As 2023, this measurement technique is considered to be one of the state-of-the-art metrics of explainability.

This program outputs a .json file containing the ROAD evaluations (using most-relevant-first, or MoRF, and least-relevant-first, or LeRF for each sample in the DataModule. Values for MoRF and LeRF represent averages by removing 20, 40, 60 and 80% of most or least relevant pixels respectively from the image, and averaging results for all these percentiles.

Note

This application is relatively slow when processing a large DataModule with many (positive) samples.

It is possible to pass one or several Python files (or names of mednet.config entry points or module names) as CONFIG arguments to the command line which contain the parameters listed below as Python variables. The options through the command-line (see below) will override the values of configuration files. You can run this command with <COMMAND> -H example_config.py to create a template config file.

mednet classify saliency completeness [OPTIONS] [CONFIG]...

Options

-m, --model <model>

Required A lightning module instance implementing the network architecture (not the weights, necessarily) to be used for inference. Currently, only supports pasa and densenet models.

-d, --datamodule <datamodule>

Required A lightning DataModule that will be asked for prediction DataLoaders. Typically, this includes all configured splits in a DataModule, however this is not a requirement. A DataModule that returns a single DataLoader for prediction (wrapped in a dictionary) is acceptable.

-o, --output-folder <output_folder>

Required Name of the folder where to store the output .json file containing all measures.

Default:

'results'

-x, --device <device>

Required A string indicating the device to use (e.g. “cpu” or “cuda:0”)

Default:

'cpu'

--cache-samples, --no-cache-samples

Required If set to True, loads the sample into memory, otherwise loads them at runtime.

Default:

False

-w, --weight <weight>

Required Path or URL to pretrained model file (.ckpt extension), corresponding to the architecture set with –model. Optionally, you may also pass a directory containing the result of a training session, in which case either the best (lowest validation) or latest model will be loaded.

-P, --parallel <parallel>

Required Use multiprocessing for data loading processing: if set to -1 (default), disables multiprocessing. Set to 0 to enable as many data processing instances as processing cores available in the system. Set to >= 1 to enable that many multiprocessing instances. Note that if you activate this option, then you must use –device=cpu, as using a GPU concurrently is not supported.

Default:

-1

-s, --saliency-map-algorithm <saliency_map_algorithm>

Saliency map algorithm to be used.

Default:

'gradcam'

Options:

ablationcam | eigencam | eigengradcam | fullgrad | gradcam | gradcamelementwise | gradcam++ | gradcamplusplus | hirescam | layercam | randomcam | scorecam | xgradcam

-C, --target-class <target_class>

This option should only be used with multiclass models. It defines the class to target for saliency estimation. Can be either set to “all” or “highest”. “highest” (the default), means only saliency maps for the class with the highest activation will be generated.

Options:

highest | all

-z, --positive-only, -Z, --no-positive-only

If set, and the model chosen has a single output (binary), then saliency maps will only be generated for samples of the positive class. This option has no effect for multiclass models.

-e, --percentile <percentile>

One or more percentiles (percent x100) integer values indicating the proportion of pixels to perturb in the original image to calculate both MoRF and LeRF scores.

Default:

20, 40, 60, 80

-S, --only-dataset <only_dataset>

If set, will only run the command for the named dataset on the provided datamodule, skipping any other dataset.

-P, --plot, --no-plot

Required If set, then also produces figures containing the plots of score histograms.

Default:

True

-v, --verbose

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

-H, --dump-config <dump_config>

Name of the config file to be generated

Arguments

CONFIG

Optional argument(s)

Examples:

  1. Calculate the ROAD scores using GradCAM, for an existing dataset configuration and stores them in .json files:

    mednet classify saliency completeness -vv pasa tbx11k-v1-healthy-vs-atb --saliency-map-algorithm="gradcam" --device="cuda" --weight=path/to/model-at-lowest-validation-loss.ckpt --output-json=path/to/completeness-scores.json
    
generate
Generate saliency maps for locations on input images that affected the

prediction.

The quality of saliency information depends on the saliency map algorithm and trained model.

It is possible to pass one or several Python files (or names of mednet.config entry points or module names) as CONFIG arguments to the command line which contain the parameters listed below as Python variables. The options through the command-line (see below) will override the values of configuration files. You can run this command with <COMMAND> -H example_config.py to create a template config file.

mednet classify saliency generate [OPTIONS] [CONFIG]...

Options

-m, --model <model>

Required A lightning module instance implementing the network architecture (not the weights, necessarily) to be used for inference. Currently, only supports pasa and densenet models.

-d, --datamodule <datamodule>

Required A lightning DataModule that will be asked for prediction data loaders. Typically, this includes all configured splits in a DataModule, however this is not a requirement. A DataModule that returns a single dataloader for prediction (wrapped in a dictionary) is acceptable.

-o, --output-folder <output_folder>

Required Directory in which to store saliency maps (created if does not exist)

Default:

'saliency-maps'

-x, --device <device>

Required A string indicating the device to use (e.g. “cpu” or “cuda:0”)

Default:

'cpu'

--cache-samples, --no-cache-samples

Required If set to True, loads the sample into memory, otherwise loads them at runtime.

Default:

False

-w, --weight <weight>

Required Path or URL to a pretrained model file (.ckpt extension), corresponding to the architecture set with –model. Optionally, you may also pass a directory containing the result of a training session, in which case either the best (lowest validation) or latest model will be loaded.

-P, --parallel <parallel>

Required Use multiprocessing for data loading: if set to -1 (default), disables multiprocessing data loading. Set to 0 to enable as many data loading instances as processing cores available in the system. Set to >= 1 to enable that many multiprocessing instances for data loading.

Default:

-1

-s, --saliency-map-algorithm <saliency_map_algorithm>

Saliency map algorithm to be used.

Default:

'gradcam'

Options:

ablationcam | eigencam | eigengradcam | fullgrad | gradcam | gradcamelementwise | gradcam++ | gradcamplusplus | hirescam | layercam | randomcam | scorecam | xgradcam

-C, --target-class <target_class>

This option should only be used with multiclass models. It defines the class to target for saliency estimation. Can be either set to “all” or “highest”. “highest” (the default), means only saliency maps for the class with the highest activation will be generated.

Options:

highest | all

-z, --positive-only, -Z, --no-positive-only

If set, and the model chosen has a single output (binary), then saliency maps will only be generated for samples of the positive class. This option has no effect for multiclass models.

-S, --only-dataset <only_dataset>

If set, will only run the command for the named dataset on the provided datamodule, skipping any other dataset.

-v, --verbose

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

-H, --dump-config <dump_config>

Name of the config file to be generated

Arguments

CONFIG

Optional argument(s)

Examples:

  1. Generate saliency maps using GradCAM for all prediction dataloaders on a DataModule, using a pre-trained DenseNet model, and saves them as numpy-pickeled objects on the output directory:

    mednet classify saliency generate -vv densenet tbx11k-v1-healthy-vs-atb --saliency-map-algorithm="gradcam" --weight=path/to/model-at-lowest-validation-loss.ckpt --output-folder=path/to/output
    
interpretability
Evaluate saliency map agreement with annotations (human

interpretability).

The evaluation happens by comparing saliency maps with ground-truth provided by any other means (typically following a manual annotation procedure).

Note

For obvious reasons, this evaluation is limited to datasets that contain built-in annotations which corroborate classification.

As a result of the evaluation, this application creates a single .json file that resembles the original DataModule, with added information containing the following measures, for each sample:

  • Proportional Energy: A measure that compares (UNthresholed) saliency maps with annotations (based on [WWD+20]). It estimates how much activation lies within the ground truth boxes compared to the total sum of the activations.

  • Average Saliency Focus: estimates how much of the ground truth bounding boxes area is covered by the activations. It is similar to the proportional energy measure in the sense that it does not need explicit thresholding.

It is possible to pass one or several Python files (or names of mednet.config entry points or module names) as CONFIG arguments to the command line which contain the parameters listed below as Python variables. The options through the command-line (see below) will override the values of configuration files. You can run this command with <COMMAND> -H example_config.py to create a template config file.

mednet classify saliency interpretability [OPTIONS] [CONFIG]...

Options

-m, --model <model>

Required A lightning module instance implementing the network architecture (not the weights, necessarily) to be used for inference. Currently, only supports pasa and densenet models.

-d, --datamodule <datamodule>

Required A lightning DataModule that will be asked for prediction data loaders. Typically, this includes all configured splits in a DataModule, however this is not a requirement. A DataModule that returns a single dataloader for prediction (wrapped in a dictionary) is acceptable.

-i, --input-folder <input_folder>

Required Path from where to load saliency maps. You can generate saliency maps with mednet classify saliency generate.

-t, --target-label <target_label>

Required The target label that will be analysed. It must match the target label that was used to generate the saliency maps provided with option --input-folder. Samples with all other labels are ignored.

Default:

1

-o, --output-folder <output_folder>

Required Path to the folder in which all measures will be saved.

Default:

'results'

-S, --only-dataset <only_dataset>

If set, will only run the command for the named dataset on the provided datamodule, skipping any other dataset.

-P, --plot, --no-plot

Required If set, then also produces figures containing the plots of score histograms.

Default:

True

-v, --verbose

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

-H, --dump-config <dump_config>

Name of the config file to be generated

Arguments

CONFIG

Optional argument(s)

Examples:

  1. Evaluate the generated saliency maps for their localization performance:

    mednet classify saliency interpretability -vv pasa tbx11k-v1-healthy-vs-atb --input-folder=parent-folder/saliencies/ --output-json=path/to/interpretability-scores.json
    
view

Generate heatmaps for input samples based on existing saliency maps.

It is possible to pass one or several Python files (or names of mednet.config entry points or module names) as CONFIG arguments to the command line which contain the parameters listed below as Python variables. The options through the command-line (see below) will override the values of configuration files. You can run this command with <COMMAND> -H example_config.py to create a template config file.

mednet classify saliency view [OPTIONS] [CONFIG]...

Options

-m, --model <model>

Required A lightning module instance implementing the network to be used for applying the necessary data transformations.

-d, --datamodule <datamodule>

Required A lightning DataModule containing the training, validation and test sets.

-i, --input-folder <input_folder>

Required Path to the directory containing the saliency maps for a specific visualization type.

-o, --output-folder <output_folder>

Required Directory in which to store the visualizations (created if does not exist)

Default:

'visualizations'

-G, --show-groundtruth, -g, --no-show-groundtruth

If set, visualizations for ground truth labels will be generated. Only works for datasets with bounding boxes.

-t, --threshold <threshold>

Required The pixel values above threshold % of max value are kept in the original saliency map. Everything else is set to zero. The value proposed on [WWD+20] is 0.2. Use this value if unsure.

Default:

0.2

-v, --verbose

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

-H, --dump-config <dump_config>

Name of the config file to be generated

Arguments

CONFIG

Optional argument(s)

Examples:

  1. Generate visualizations in the form of heatmaps from existing saliency maps for a dataset configuration:

    mednet classify saliency view -vv pasa tbx11k-v1-healthy-vs-atb --input-folder=parent_folder/gradcam/ --output-folder=path/to/visualizations
    

config

Command for listing, describing and copying configuration resources.

mednet config [OPTIONS] COMMAND [ARGS]...

copy

Copy a specific configuration resource so it can be modified locally.

mednet config copy [OPTIONS] SOURCE DESTINATION

Options

-v, --verbose

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

Arguments

SOURCE

Required argument

DESTINATION

Required argument

Examples:

1. Make a copy of one of the stock configuration files locally, so it can be
adapted:
$ mednet config copy montgomery -vvv newdataset.py

describe

Describe a specific configuration file.

mednet config describe [OPTIONS] NAME...

Options

-v, --verbose

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

Arguments

NAME

Required argument(s)

Examples:

1. Describe the Montgomery dataset configuration:
mednet config describe montgomery
2. Describe the Montgomery dataset configuration and lists its
contents:
mednet config describe montgomery -v

list

List configuration files installed.

mednet config list [OPTIONS]

Options

-v, --verbose

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

Examples:

1. Lists all configuration resources installed:
mednet config list
2. Lists all configuration resources and their descriptions (notice this may
be slow as it needs to load all modules once):
mednet config list -v

database

Command for listing and verifying databases installed.

mednet database [OPTIONS] COMMAND [ARGS]...

check

Check file access on a database configuration split.

mednet database check [OPTIONS] ENTRYPOINT

Options

-l, --limit <limit>

Required Limit check to the first N samples in each split in the configuration, making the check sensibly faster. Set it to zero (default) to check everything.

Default:

0

-v, --verbose

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

Arguments

ENTRYPOINT

Required argument

Examples:

  1. Check if all files from the config split ‘montgomery-f0’ of the Montgomery database can be loaded:

    mednet database check -vv montgomery-f0
    

list

List all supported and configured databases.

mednet database list [OPTIONS]

Options

-v, --verbose

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

Examples:

1. To install a database, set up its data directory (“datadir”). For
example, to setup access to Montgomery files you downloaded locally at
the directory “/path/to/montgomery/files”, edit the RC file (typically
$HOME/.config/mednet.toml), and add a line like the following:
[datadir]
montgomery = "/path/to/montgomery/files"

Note

This setting is case-sensitive.

2. List all raw databases supported (and configured):
$ mednet database list

preprocess

Run pre-processing on databases using a set of model transforms.

This command can be used to generate more compact versions of the databases that can be used to speed-up data loading. This is done by running both data loading and model transforms so that stored samples are small enough, and require no transformations when loading.

It is possible to pass one or several Python files (or names of mednet.config entry points or module names) as CONFIG arguments to the command line which contain the parameters listed below as Python variables. The options through the command-line (see below) will override the values of configuration files. You can run this command with <COMMAND> -H example_config.py to create a template config file.

mednet database preprocess [OPTIONS] [CONFIG]...

Options

-o, --output-folder <output_folder>

Required Directory in which to store results (created if does not exist)

Default:

'results'

-m, --model <model>

Required A lightning module instance implementing the network to be trained

-d, --datamodule <datamodule>

Required A lightning DataModule containing the training and validation sets.

-l, --limit <limit>

Required Limit preprocessing to the first N samples in each split in the configuration, making the job sensibly faster. Set it to zero (default) to preprocess everything.

Default:

0

-g, --grayscale

For images to be grayscale, if set.

Default:

False

-v, --verbose

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

-H, --dump-config <dump_config>

Name of the config file to be generated

Arguments

CONFIG

Optional argument(s)

Examples:

  1. Pre-process the Montgomery (classification) dataset using model transforms from the pasa model.

    mednet database preprocess -vv montgomery pasa
    
  2. Pre-process the CXR8 (segmentation) dataset using model transforms from the lwnet model.

    mednet database preprocess -vv cxr8 lwnet
    

detect

Object detection specialized commands.

mednet detect [OPTIONS] COMMAND [ARGS]...

evaluate

Evaluate predictions (from a model) on a classification task.

It is possible to pass one or several Python files (or names of mednet.config entry points or module names) as CONFIG arguments to the command line which contain the parameters listed below as Python variables. The options through the command-line (see below) will override the values of configuration files. You can run this command with <COMMAND> -H example_config.py to create a template config file.

mednet detect evaluate [OPTIONS] [CONFIG]...

Options

-p, --predictions <predictions>

Required Filename in which predictions are currently stored

-o, --output-folder <output_folder>

Required Directory in which to store results (created if does not exist)

Default:

'results'

-t, --iou-threshold <iou_threshold>

IOU threshold by which we consider successful object detection.

-b, --binning <binning>

Required The binning algorithm to use for computing the bin widths and distribution for histograms. Choose from algorithms supported by numpy.histogram(), or a simple integer indicating the number of bins to have in the interval [0, 1].

Default:

'50'

-P, --plot, --no-plot

Required If set, then also produces figures containing the plots of performance curves and score histograms.

Default:

True

-v, --verbose

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

-H, --dump-config <dump_config>

Name of the config file to be generated

Arguments

CONFIG

Optional argument(s)

Examples:

  1. Run evaluation on an existing prediction output:

    mednet detect evaluate -vv --predictions=path/to/predictions.json
    

experiment

Run a complete experiment, from training, to prediction and evaluation.

Accepts all of command-line options from the train script.

This script is just a wrapper around the individual scripts for training, running prediction, and evaluating. It organises the output in a preset way:

└─ <output-folder>/  # the generated model will be here
   ├── predictions.json  # the prediction outputs
   └── evaluation.json  # the evaluation outputs

Note that complete experiments do not have options related to specific task functionality. Only the most generic set of options is available. To execute specific tasks, use the specialized command-line interfaces available in task-specific command groups.

It is possible to pass one or several Python files (or names of mednet.config entry points or module names) as CONFIG arguments to the command line which contain the parameters listed below as Python variables. The options through the command-line (see below) will override the values of configuration files. You can run this command with <COMMAND> -H example_config.py to create a template config file.

mednet experiment [OPTIONS] [CONFIG]...

Options

-o, --output-folder <output_folder>

Required Directory in which to store results (created if does not exist)

Default:

'results'

-m, --model <model>

Required A lightning module instance implementing the network to be trained

-d, --datamodule <datamodule>

Required A lightning DataModule containing the training and validation sets.

-b, --batch-size <batch_size>

Required Number of samples in every batch (this parameter affects memory requirements for the network). If the number of samples in the batch is larger than the total number of samples available for training, this value is truncated. If this number is smaller, then batches of the specified size are created and fed to the network until there are no more new samples to feed (epoch is finished). If the total number of training samples is not a multiple of the batch-size, the last batch will be smaller than the first, unless –drop-incomplete-batch is set, in which case this batch is not used.

Default:

1

-a, --accumulate-grad-batches <accumulate_grad_batches>

Required Number of accumulations for backward propagation to accumulate “ gradients over k batches before stepping the optimizer. This parameter, used in conjunction with the batch-size, may be used to reduce the number of samples loaded in each iteration, to affect memory usage in exchange for processing time (more iterations). This is useful interesting when one is training on GPUs with a limited amount of onboard RAM. The default of 1 forces the whole batch to be processed at once. Otherwise the batch is multiplied by accumulate-grad-batches pieces, and gradients are accumulated to complete each training step.

Default:

1

-D, --drop-incomplete-batch, --no-drop-incomplete-batch

Required If set, the last batch in an epoch will be dropped if incomplete. If you set this option, you should also consider increasing the total number of epochs of training, as the total number of training steps may be reduced.

Default:

False

-e, --epochs <epochs>

Required Number of epochs (complete training set passes) to train for. If continuing from a saved checkpoint, ensure to provide a greater number of epochs than was saved in the checkpoint to be loaded.

Default:

1000

-p, --validation-period <validation_period>

Required Number of epochs after which validation happens. By default, we run validation after every training epoch (period=1). You can change this to make validation more sparse, by increasing the validation period. Notice that this affects checkpoint saving. While checkpoints are created after every training step (the last training step always triggers the overriding of latest checkpoint), and this process is independent of validation runs, evaluation of the ‘best’ model obtained so far based on those will be influenced by this setting.

Default:

1

-x, --device <device>

Required A string indicating the device to use (e.g. “cpu”, “cuda”, “mps”, “cuda:0”). Can be passed as a string, or as an environment variable (e.g. MEDNET_DEVICE=cpu or MEDNET_DEVICE=mps).

Default:

'cpu'

--cache-samples, --no-cache-samples

Required If set to True, loads the sample into memory, otherwise loads them at runtime.

Default:

False

-s, --seed <seed>

Seed to use for the random number generator

Default:

42

-P, --parallel <parallel>

Required Use multiprocessing for data loading: if set to -1 (default), disables multiprocessing data loading. Set to 0 to enable as many data loading instances as processing cores available in the system. Set to >= 1 to enable that many multiprocessing instances for data loading.

Default:

-1

-I, --monitoring-interval <monitoring_interval>

Required Time between checks for the use of resources during each training epoch, in seconds. An interval of 5 seconds, for example, will lead to CPU and GPU resources being probed every 5 seconds during each training epoch. Values registered in the training logs correspond to averages (or maxima) observed through possibly many probes in each epoch. Notice that setting a very small value may cause the probing process to become extremely busy, potentially biasing the overall perception of resource usage.

Default:

5.0

-A, --augmentations <augmentations>

Models that can be trained in this package are shipped without explicit data augmentations. This option allows you to define a list of data augmentations to use for training the selected model.

-B, --balance-classes, -N, --no-balance-classes

Required If set, balances the loss term to take into consideration the occurence of each target in the training and validation splits.

Default:

True

-W, --initial-weights <initial_weights>

Path or URL to pretrained model file (.ckpt extension), corresponding to the architecture set with –model. Optionally, you may also pass a directory containing the result of a training session, in which case either the best (lowest validation) or latest model will be loaded. Note these weights will be overwritten when re-starting training from a checkpoint. This parameter should only be used as a set of initial weights (when starting a new training session).

-v, --verbose

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

-H, --dump-config <dump_config>

Name of the config file to be generated

Arguments

CONFIG

Optional argument(s)

Environment variables

MEDNET_DEVICE

Provide a default for --device

Examples:

1. Train a lwnet model with drive dataset, on the CPU, for only two
epochs, then runs prediction (inference) and evaluation on listed datasets,
report performance as a table and figures:
$ mednet experiment -vv lwnet drive --epochs=2

info

Provide information about the current installation.

mednet info [OPTIONS]

Options

-v, --verbose

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

Examples:

  1. Provide information about the current installation:

    mednet info
    

predict

Run inference on input samples, using a pre-trained model.

It is possible to pass one or several Python files (or names of mednet.config entry points or module names) as CONFIG arguments to the command line which contain the parameters listed below as Python variables. The options through the command-line (see below) will override the values of configuration files. You can run this command with <COMMAND> -H example_config.py to create a template config file.

mednet predict [OPTIONS] [CONFIG]...

Options

-o, --output-folder <output_folder>

Required Directory in which to save predictions (created if does not exist)

Default:

'results'

-m, --model <model>

Required A lightning module instance implementing the network architecture (not the weights, necessarily) to be used for prediction.

-d, --datamodule <datamodule>

Required A lightning DataModule that will be asked for prediction data loaders. Typically, this includes all configured splits in a DataModule, however this is not a requirement. A DataModule that returns a single dataloader for prediction (wrapped in a dictionary) is acceptable.

-b, --batch-size <batch_size>

Required Number of samples in every batch (this parameter affects memory requirements for the network).

Default:

1

-d, --device <device>

Required A string indicating the device to use (e.g. “cpu”, “cuda”, “mps”, “cuda:0”). Can be passed as a string, or as an environment variable (e.g. MEDNET_DEVICE=cpu or MEDNET_DEVICE=mps).

Default:

'cpu'

-w, --weight <weight>

Required Path or URL to pretrained model file (.ckpt extension), corresponding to the architecture set with –model. Optionally, you may also pass a directory containing the result of a training session, in which case either the best (lowest validation) or latest model will be loaded.

-P, --parallel <parallel>

Required Use multiprocessing for data loading: if set to -1 (default), disables multiprocessing data loading. Set to 0 to enable as many data loading instances as processing cores available in the system. Set to >= 1 to enable that many multiprocessing instances for data loading.

Default:

-1

-v, --verbose

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

-H, --dump-config <dump_config>

Name of the config file to be generated

Arguments

CONFIG

Optional argument(s)

Environment variables

MEDNET_DEVICE

Provide a default for --device

Examples:

  1. Run prediction on an existing DataModule configuration:

    mednet predict -vv lwnet drive --weight=path/to/model.ckpt --output-folder=path/to/predictions
    
  2. Enable multi-processing data loading with 6 processes:

    mednet predict -vv lwnet drive --parallel=6 --weight=path/to/model.ckpt --output-folder=path/to/predictions
    

segment

Image segmentation specialized commands.

mednet segment [OPTIONS] COMMAND [ARGS]...

dump-annotations

Dump annotations in a given folder, after pre-processing.

It is possible to pass one or several Python files (or names of mednet.config entry points or module names) as CONFIG arguments to the command line which contain the parameters listed below as Python variables. The options through the command-line (see below) will override the values of configuration files. You can run this command with <COMMAND> -H example_config.py to create a template config file.

mednet segment dump-annotations [OPTIONS] [CONFIG]...

Options

-o, --output-folder <output_folder>

Required Directory in which to save predictions (created if does not exist)

Default:

'predictions'

-m, --model <model>

Required A lightning module instance that will be used to retrieve pre-processing transforms.

-d, --datamodule <datamodule>

Required A lightning DataModule that will be asked for prediction data loaders. Typically, this includes all configured splits in a DataModule, however this is not a requirement. A DataModule that returns a single dataloader for prediction (wrapped in a dictionary) is acceptable.

-v, --verbose

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

-H, --dump-config <dump_config>

Name of the config file to be generated

Arguments

CONFIG

Optional argument(s)

Examples:

  1. Dump annotations for a dataset after pre-processing on a particular directory:

    mednet segment dump-annotations -vv lwnet drive-2nd --output-folder=path/to/annotations
    

evaluate

Evaluate predictions (from a model) on a segmentation task.

It is possible to pass one or several Python files (or names of mednet.config entry points or module names) as CONFIG arguments to the command line which contain the parameters listed below as Python variables. The options through the command-line (see below) will override the values of configuration files. You can run this command with <COMMAND> -H example_config.py to create a template config file.

mednet segment evaluate [OPTIONS] [CONFIG]...

Options

-p, --predictions <predictions>

Required Path to the JSON file describing available predictions. The actual predictions are supposed to lie on the same folder.

-o, --output-folder <output_folder>

Required Directory in which to store results (created if does not exist)

Default:

'results'

-t, --threshold <threshold>

This number is used to define positives and negatives from probability maps, and used to report metrics based on a threshold chosen a priori. It can be set to a floating-point value, or to the name of dataset split in --predictions.

Default:

'0.5'

-m, --metric <metric>

Required If threshold is set to the name of a split in --predictions, then this parameter defines the metric function to be used to evaluate the threshold at which the metric reaches its maximum value. All other splits are evaluated with respect to this threshold.

Default:

'f1'

Options:

precision | recall | specificity | accuracy | jaccard | f1

-s, --steps <steps>

Required Number of steps for evaluating metrics on various splits. This value is used when drawing precision-recall plots, or when deciding the highest metric value on splits.

Default:

100

-a, --compare-annotator <compare_annotator>

Path to a JSON file as produced by the CLI dump-annotations, containing splits and sample lists with associated HDF5 files where we can find pre-processed annotation masks. These annotations will be compared with the target annotations on the main predictions. In this case, a row is added for each available split in the evaluation table.

-P, --plot, --no-plot

Required If set, then also produces figures containing the plots of performance curves and score histograms.

Default:

True

-v, --verbose

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

-H, --dump-config <dump_config>

Name of the config file to be generated

Arguments

CONFIG

Optional argument(s)

Examples:

1. Runs evaluation from existing predictions:
$ mednet segment evaluate -vv --predictions=path/to/predictions.json --output-folder=path/to/results
  1. Runs evaluation from existing predictions, incorporate a second set of annotations into the evaluation:

    $ mednet segment evaluate -vv --predictions=path/to/predictions.json --compare-annotator=path/to/annotations.json --output-folder=path/to/results
    

view

Display predictions generated by a model.

It is possible to pass one or several Python files (or names of mednet.config entry points or module names) as CONFIG arguments to the command line which contain the parameters listed below as Python variables. The options through the command-line (see below) will override the values of configuration files. You can run this command with <COMMAND> -H example_config.py to create a template config file.

mednet segment view [OPTIONS] [CONFIG]...

Options

-p, --predictions <predictions>

Required Path to the JSON file describing available predictions. The actual predictions are supposed to lie on the same folder.

-o, --output-folder <output_folder>

Required Directory in which to store results (created if does not exist)

-t, --threshold <threshold>

This number is used to define positives and negatives from probability maps, and used to report metrics based on a threshold chosen a priori. It can be set to a floating-point value, or to the name of dataset split in --predictions.

Default:

'0.5'

-m, --metric <metric>

Required If threshold is set to the name of a split in --predictions, then this parameter defines the metric function to be used to evaluate the threshold at which the metric reaches its maximum value. All other splits are evaluated with respect to this threshold.

Default:

'f1'

Options:

precision | recall | specificity | accuracy | jaccard | f1

-s, --steps <steps>

Required Number of steps for evaluating metrics on various splits. This value is used only when evaluating thresholds on a datamodule split.

Default:

100

-e, --show-errors, -E, --no-show-errors

Required If set, then shows a colorized version of the segmentation map in which false-positives are marked in red, and false-negatives in green. True positives are always marked in white.

Default:

False

-a, --alpha <alpha>

Required Defines the transparency weighting between the original image and the predicted segmentation maps. A value of 1.0 makes the program output only segmentation maps. A value of 0.0 makes the program output only the processed image.

Default:

0.6

-v, --verbose

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

-H, --dump-config <dump_config>

Name of the config file to be generated

Arguments

CONFIG

Optional argument(s)

Examples:

1. Produces visualisations for the given predictions:
$ mednet segment view -vv --predictions=path/to/predictions.json --output-folder=path/to/results
  1. Produces visualisations for the given (dumped) annotations:

    $ mednet segment view -vv --predictions=path/to/annotations.json --output-folder=path/to/results
    

train

Train a model on a given datamodule (task-specific).

Training is performed for a configurable number of epochs, and generates checkpoints. Checkpoints are model files with a .ckpt extension that are used in subsequent tasks or from which training can be resumed.

It is possible to pass one or several Python files (or names of mednet.config entry points or module names) as CONFIG arguments to the command line which contain the parameters listed below as Python variables. The options through the command-line (see below) will override the values of configuration files. You can run this command with <COMMAND> -H example_config.py to create a template config file.

mednet train [OPTIONS] [CONFIG]...

Options

-o, --output-folder <output_folder>

Required Directory in which to store results (created if does not exist)

Default:

'results'

-m, --model <model>

Required A lightning module instance implementing the network to be trained

-d, --datamodule <datamodule>

Required A lightning DataModule containing the training and validation sets.

-b, --batch-size <batch_size>

Required Number of samples in every batch (this parameter affects memory requirements for the network). If the number of samples in the batch is larger than the total number of samples available for training, this value is truncated. If this number is smaller, then batches of the specified size are created and fed to the network until there are no more new samples to feed (epoch is finished). If the total number of training samples is not a multiple of the batch-size, the last batch will be smaller than the first, unless –drop-incomplete-batch is set, in which case this batch is not used.

Default:

1

-a, --accumulate-grad-batches <accumulate_grad_batches>

Required Number of accumulations for backward propagation to accumulate “ gradients over k batches before stepping the optimizer. This parameter, used in conjunction with the batch-size, may be used to reduce the number of samples loaded in each iteration, to affect memory usage in exchange for processing time (more iterations). This is useful interesting when one is training on GPUs with a limited amount of onboard RAM. The default of 1 forces the whole batch to be processed at once. Otherwise the batch is multiplied by accumulate-grad-batches pieces, and gradients are accumulated to complete each training step.

Default:

1

-D, --drop-incomplete-batch, --no-drop-incomplete-batch

Required If set, the last batch in an epoch will be dropped if incomplete. If you set this option, you should also consider increasing the total number of epochs of training, as the total number of training steps may be reduced.

Default:

False

-e, --epochs <epochs>

Required Number of epochs (complete training set passes) to train for. If continuing from a saved checkpoint, ensure to provide a greater number of epochs than was saved in the checkpoint to be loaded.

Default:

1000

-p, --validation-period <validation_period>

Required Number of epochs after which validation happens. By default, we run validation after every training epoch (period=1). You can change this to make validation more sparse, by increasing the validation period. Notice that this affects checkpoint saving. While checkpoints are created after every training step (the last training step always triggers the overriding of latest checkpoint), and this process is independent of validation runs, evaluation of the ‘best’ model obtained so far based on those will be influenced by this setting.

Default:

1

-x, --device <device>

Required A string indicating the device to use (e.g. “cpu”, “cuda”, “mps”, “cuda:0”). Can be passed as a string, or as an environment variable (e.g. MEDNET_DEVICE=cpu or MEDNET_DEVICE=mps).

Default:

'cpu'

--cache-samples, --no-cache-samples

Required If set to True, loads the sample into memory, otherwise loads them at runtime.

Default:

False

-s, --seed <seed>

Seed to use for the random number generator

Default:

42

-P, --parallel <parallel>

Required Use multiprocessing for data loading: if set to -1 (default), disables multiprocessing data loading. Set to 0 to enable as many data loading instances as processing cores available in the system. Set to >= 1 to enable that many multiprocessing instances for data loading.

Default:

-1

-I, --monitoring-interval <monitoring_interval>

Required Time between checks for the use of resources during each training epoch, in seconds. An interval of 5 seconds, for example, will lead to CPU and GPU resources being probed every 5 seconds during each training epoch. Values registered in the training logs correspond to averages (or maxima) observed through possibly many probes in each epoch. Notice that setting a very small value may cause the probing process to become extremely busy, potentially biasing the overall perception of resource usage.

Default:

5.0

-A, --augmentations <augmentations>

Models that can be trained in this package are shipped without explicit data augmentations. This option allows you to define a list of data augmentations to use for training the selected model.

-B, --balance-classes, -N, --no-balance-classes

Required If set, balances the loss term to take into consideration the occurence of each target in the training and validation splits.

Default:

True

-W, --initial-weights <initial_weights>

Path or URL to pretrained model file (.ckpt extension), corresponding to the architecture set with –model. Optionally, you may also pass a directory containing the result of a training session, in which case either the best (lowest validation) or latest model will be loaded. Note these weights will be overwritten when re-starting training from a checkpoint. This parameter should only be used as a set of initial weights (when starting a new training session).

-v, --verbose

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

-H, --dump-config <dump_config>

Name of the config file to be generated

Arguments

CONFIG

Optional argument(s)

Environment variables

MEDNET_DEVICE

Provide a default for --device

Examples:

  1. Train a Pasa model with the montgomery dataset (classification task):

    mednet train -vv pasa montgomery
    
  2. Train a Little WNet model with the drive dataset (vessel segmentation task):

    mednet train -vv lwnet drive
    
  3. Train a Densenet model with the shenzhen dataset, while starting from a set of pre-trained weights:

    mednet train -vv densenet shenzhen --initial-weights=path/to/checkpoint.ckpt
    

train-analysis

Create a plot for each metric in the training logs and saves them in a .pdf file.

mednet train-analysis [OPTIONS]

Options

-l, --logdir <logdir>

Required Path to the directory containing the Tensorboard training logs

-o, --output-folder <output_folder>

Required Directory in which to store results (created if does not exist)

Default:

'results'

-v, --verbose

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

Examples:

1. Analyze a training log and produces various plots:
mednet train-analysis -vv -l results/logs

upload

Upload results from a classification experiment folder to GitLab’s MLFlow server.

It is possible to pass one or several Python files (or names of mednet.config entry points or module names) as CONFIG arguments to the command line which contain the parameters listed below as Python variables. The options through the command-line (see below) will override the values of configuration files. You can run this command with <COMMAND> -H example_config.py to create a template config file.

mednet upload [OPTIONS] [CONFIG]...

Options

-p, --project-path <project_path>

Required Path to the project where to upload model entries

Default:

'medai/software/mednet'

-e, --experiment-folder <experiment_folder>

Required Directory in which to upload results from

Default:

'results'

-n, --experiment-name <experiment_name>

Required A string indicating the experiment name (e.g. “exp-pasa-mc” or “exp-densenet-mc-ch”)

-r, --run-name <run_name>

Required A string indicating the run name (e.g. “run-1”)

-l, --upload-limit-mb <upload_limit_mb>

Required Maximim upload size in MB (set to 0 for no limit).

Default:

10

-v, --verbose

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

-H, --dump-config <dump_config>

Name of the config file to be generated

Arguments

CONFIG

Optional argument(s)

Examples:

  1. Upload an existing experiment result from a path it resides on (with a default experiment name as {model-name}_{database-name} and a default run name as {date-time}):

    mednet upload --experiment-folder=/path/to/results
    
  2. Upload an existing experiment result with an experiment name:

    mednet upload --experiment-folder=/path/to/results --experiment-name=exp-pasa_mc
    
  3. Upload an existing experiment result with a run name:

    mednet upload --experiment-folder=/path/to/results --run-name=run-1
    
  4. Upload an existing experiment result with defining a size limit of 20MB for each file:

    mednet upload --experiment-folder=/path/to/results --upload-limit-mb=20
    

version

Provide version of the current installation.

mednet version [OPTIONS]

Options

-v, --verbose

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

Examples:

  1. Provide versio of the current installation:

    mednet version