Command-line Interface¶
This section contains an overview of command-line applications shipped with this package.
mednet¶
Medical image analysis AI toolbox.
mednet [OPTIONS] COMMAND [ARGS]...
classify¶
Image classification specialized commands.
mednet classify [OPTIONS] COMMAND [ARGS]...
evaluate¶
Evaluate predictions (from a model) on a classification task.
It is possible to pass one or several Python
files (or names of mednet.config
entry points or module names) as
CONFIG arguments to the command line which contain the parameters listed below as Python variables. The options through the command-line (see below)
will override the values of configuration files. You can run this command with
<COMMAND> -H example_config.py
to create a template config file.
mednet classify evaluate [OPTIONS] [CONFIG]...
Options
- -p, --predictions <predictions>¶
Required Filename in which predictions are currently stored
- -o, --output-folder <output_folder>¶
Required Directory in which to store results (created if does not exist)
- Default:
'results'
- -t, --threshold <threshold>¶
Required This value is used to define positives and negatives from probability outputs in predictions, and report performance measures on binary classification tasks. It should either come from the training set or a separate validation set to avoid biasing the analysis. Optionally, if you provide a multi-split set of predictions as input, this may also be the name of an existing split (e.g.
validation
) from which the threshold will be estimated (by calculating the threshold leading to the highest F1-score on that set) and then applied to the subsequent sets. This value is not used for multi-class classification tasks.- Default:
0.5
- -b, --binning <binning>¶
Required The binning algorithm to use for computing the bin widths and distribution for histograms. Choose from algorithms supported by
numpy.histogram()
, or a simple integer indicating the number of bins to have in the interval[0, 1]
.- Default:
'50'
- -P, --plot, --no-plot¶
Required If set, then also produces figures containing the plots of performance curves and score histograms.
- Default:
True
- -I, --credible-regions, --no-credible-regions¶
Required If set, then evaluates also credible intervals. Notice the evaluation of ROC-AUC and Average Precision confidence margins can be rather slow for larger datasets.
- Default:
False
- -s, --seed <seed>¶
Seed to use for the random number generator (used when doing Monte Carlo ” simulations required for the evaluation of credible regions for F1-score).
- Default:
42
- -v, --verbose¶
Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).
- Default:
0
- -H, --dump-config <dump_config>¶
Name of the config file to be generated
Arguments
- CONFIG¶
Optional argument(s)
Examples:
Run evaluation on an existing prediction output:
mednet classify evaluate -vv --predictions=path/to/predictions.json
Run evaluation on an existing prediction output, tune threshold a priori on the validation set:
mednet classify evaluate -vv --predictions=path/to/predictions.json --threshold=validation
saliency¶
Generate, evaluate and view saliency maps.
mednet classify saliency [OPTIONS] COMMAND [ARGS]...
completeness¶
- Evaluate saliency map algorithm completeness using RemOve And Debias
(ROAD).
For the selected saliency map algorithm, evaluates the completeness of explanations using the RemOve And Debias (ROAD) algorithm. The ROAD algorithm was first described at [RLB+22]. It estimates explainability (in the completeness sense) of saliency mapping algorithms by substituting relevant pixels in the input image by a local average, re-running prediction on the altered image, and measuring changes in the output classification score when said perturbations are in place. By substituting most or least relevant pixels with surrounding averages, the ROAD algorithm estimates the importance of such elements in the produced saliency map. As 2023, this measurement technique is considered to be one of the state-of-the-art metrics of explainability.
This program outputs a .json file containing the ROAD evaluations (using most-relevant-first, or MoRF, and least-relevant-first, or LeRF for each sample in the DataModule. Values for MoRF and LeRF represent averages by removing 20, 40, 60 and 80% of most or least relevant pixels respectively from the image, and averaging results for all these percentiles.
Note
This application is relatively slow when processing a large DataModule with many (positive) samples.
It is possible to pass one or several Python
files (or names of mednet.config
entry points or module names) as
CONFIG arguments to the command line which contain the parameters listed below as Python variables. The options through the command-line (see below)
will override the values of configuration files. You can run this command with
<COMMAND> -H example_config.py
to create a template config file.
mednet classify saliency completeness [OPTIONS] [CONFIG]...
Options
- -m, --model <model>¶
Required A lightning module instance implementing the network architecture (not the weights, necessarily) to be used for inference. Currently, only supports pasa and densenet models.
- -d, --datamodule <datamodule>¶
Required A lightning DataModule that will be asked for prediction DataLoaders. Typically, this includes all configured splits in a DataModule, however this is not a requirement. A DataModule that returns a single DataLoader for prediction (wrapped in a dictionary) is acceptable.
- -o, --output-folder <output_folder>¶
Required Name of the folder where to store the output .json file containing all measures.
- Default:
'results'
- -x, --device <device>¶
Required A string indicating the device to use (e.g. “cpu” or “cuda:0”)
- Default:
'cpu'
- --cache-samples, --no-cache-samples¶
Required If set to True, loads the sample into memory, otherwise loads them at runtime.
- Default:
False
- -w, --weight <weight>¶
Required Path or URL to pretrained model file (.ckpt extension), corresponding to the architecture set with –model. Optionally, you may also pass a directory containing the result of a training session, in which case either the best (lowest validation) or latest model will be loaded.
- -P, --parallel <parallel>¶
Required Use multiprocessing for data loading processing: if set to -1 (default), disables multiprocessing. Set to 0 to enable as many data processing instances as processing cores available in the system. Set to >= 1 to enable that many multiprocessing instances. Note that if you activate this option, then you must use –device=cpu, as using a GPU concurrently is not supported.
- Default:
-1
- -s, --saliency-map-algorithm <saliency_map_algorithm>¶
Saliency map algorithm to be used.
- Default:
'gradcam'
- Options:
ablationcam | eigencam | eigengradcam | fullgrad | gradcam | gradcamelementwise | gradcam++ | gradcamplusplus | hirescam | layercam | randomcam | scorecam | xgradcam
- -C, --target-class <target_class>¶
This option should only be used with multiclass models. It defines the class to target for saliency estimation. Can be either set to “all” or “highest”. “highest” (the default), means only saliency maps for the class with the highest activation will be generated.
- Options:
highest | all
- -z, --positive-only, -Z, --no-positive-only¶
If set, and the model chosen has a single output (binary), then saliency maps will only be generated for samples of the positive class. This option has no effect for multiclass models.
- -e, --percentile <percentile>¶
One or more percentiles (percent x100) integer values indicating the proportion of pixels to perturb in the original image to calculate both MoRF and LeRF scores.
- Default:
20, 40, 60, 80
- -S, --only-dataset <only_dataset>¶
If set, will only run the command for the named dataset on the provided datamodule, skipping any other dataset.
- -P, --plot, --no-plot¶
Required If set, then also produces figures containing the plots of score histograms.
- Default:
True
- -v, --verbose¶
Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).
- Default:
0
- -H, --dump-config <dump_config>¶
Name of the config file to be generated
Arguments
- CONFIG¶
Optional argument(s)
Examples:
Calculate the ROAD scores using GradCAM, for an existing dataset configuration and stores them in .json files:
mednet classify saliency completeness -vv pasa tbx11k-v1-healthy-vs-atb --saliency-map-algorithm="gradcam" --device="cuda" --weight=path/to/model-at-lowest-validation-loss.ckpt --output-json=path/to/completeness-scores.json
generate¶
- Generate saliency maps for locations on input images that affected the
prediction.
The quality of saliency information depends on the saliency map algorithm and trained model.
It is possible to pass one or several Python
files (or names of mednet.config
entry points or module names) as
CONFIG arguments to the command line which contain the parameters listed below as Python variables. The options through the command-line (see below)
will override the values of configuration files. You can run this command with
<COMMAND> -H example_config.py
to create a template config file.
mednet classify saliency generate [OPTIONS] [CONFIG]...
Options
- -m, --model <model>¶
Required A lightning module instance implementing the network architecture (not the weights, necessarily) to be used for inference. Currently, only supports pasa and densenet models.
- -d, --datamodule <datamodule>¶
Required A lightning DataModule that will be asked for prediction data loaders. Typically, this includes all configured splits in a DataModule, however this is not a requirement. A DataModule that returns a single dataloader for prediction (wrapped in a dictionary) is acceptable.
- -o, --output-folder <output_folder>¶
Required Directory in which to store saliency maps (created if does not exist)
- Default:
'saliency-maps'
- -x, --device <device>¶
Required A string indicating the device to use (e.g. “cpu” or “cuda:0”)
- Default:
'cpu'
- --cache-samples, --no-cache-samples¶
Required If set to True, loads the sample into memory, otherwise loads them at runtime.
- Default:
False
- -w, --weight <weight>¶
Required Path or URL to a pretrained model file (.ckpt extension), corresponding to the architecture set with –model. Optionally, you may also pass a directory containing the result of a training session, in which case either the best (lowest validation) or latest model will be loaded.
- -P, --parallel <parallel>¶
Required Use multiprocessing for data loading: if set to -1 (default), disables multiprocessing data loading. Set to 0 to enable as many data loading instances as processing cores available in the system. Set to >= 1 to enable that many multiprocessing instances for data loading.
- Default:
-1
- -s, --saliency-map-algorithm <saliency_map_algorithm>¶
Saliency map algorithm to be used.
- Default:
'gradcam'
- Options:
ablationcam | eigencam | eigengradcam | fullgrad | gradcam | gradcamelementwise | gradcam++ | gradcamplusplus | hirescam | layercam | randomcam | scorecam | xgradcam
- -C, --target-class <target_class>¶
This option should only be used with multiclass models. It defines the class to target for saliency estimation. Can be either set to “all” or “highest”. “highest” (the default), means only saliency maps for the class with the highest activation will be generated.
- Options:
highest | all
- -z, --positive-only, -Z, --no-positive-only¶
If set, and the model chosen has a single output (binary), then saliency maps will only be generated for samples of the positive class. This option has no effect for multiclass models.
- -S, --only-dataset <only_dataset>¶
If set, will only run the command for the named dataset on the provided datamodule, skipping any other dataset.
- -v, --verbose¶
Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).
- Default:
0
- -H, --dump-config <dump_config>¶
Name of the config file to be generated
Arguments
- CONFIG¶
Optional argument(s)
Examples:
Generate saliency maps using GradCAM for all prediction dataloaders on a DataModule, using a pre-trained DenseNet model, and saves them as numpy-pickeled objects on the output directory:
mednet classify saliency generate -vv densenet tbx11k-v1-healthy-vs-atb --saliency-map-algorithm="gradcam" --weight=path/to/model-at-lowest-validation-loss.ckpt --output-folder=path/to/output
interpretability¶
- Evaluate saliency map agreement with annotations (human
interpretability).
The evaluation happens by comparing saliency maps with ground-truth provided by any other means (typically following a manual annotation procedure).
Note
For obvious reasons, this evaluation is limited to datasets that contain built-in annotations which corroborate classification.
As a result of the evaluation, this application creates a single .json file that resembles the original DataModule, with added information containing the following measures, for each sample:
Proportional Energy: A measure that compares (UNthresholed) saliency maps with annotations (based on [WWD+20]). It estimates how much activation lies within the ground truth boxes compared to the total sum of the activations.
Average Saliency Focus: estimates how much of the ground truth bounding boxes area is covered by the activations. It is similar to the proportional energy measure in the sense that it does not need explicit thresholding.
It is possible to pass one or several Python
files (or names of mednet.config
entry points or module names) as
CONFIG arguments to the command line which contain the parameters listed below as Python variables. The options through the command-line (see below)
will override the values of configuration files. You can run this command with
<COMMAND> -H example_config.py
to create a template config file.
mednet classify saliency interpretability [OPTIONS] [CONFIG]...
Options
- -m, --model <model>¶
Required A lightning module instance implementing the network architecture (not the weights, necessarily) to be used for inference. Currently, only supports pasa and densenet models.
- -d, --datamodule <datamodule>¶
Required A lightning DataModule that will be asked for prediction data loaders. Typically, this includes all configured splits in a DataModule, however this is not a requirement. A DataModule that returns a single dataloader for prediction (wrapped in a dictionary) is acceptable.
- -i, --input-folder <input_folder>¶
Required Path from where to load saliency maps. You can generate saliency maps with
mednet classify saliency generate
.
- -t, --target-label <target_label>¶
Required The target label that will be analysed. It must match the target label that was used to generate the saliency maps provided with option
--input-folder
. Samples with all other labels are ignored.- Default:
1
- -o, --output-folder <output_folder>¶
Required Path to the folder in which all measures will be saved.
- Default:
'results'
- -S, --only-dataset <only_dataset>¶
If set, will only run the command for the named dataset on the provided datamodule, skipping any other dataset.
- -P, --plot, --no-plot¶
Required If set, then also produces figures containing the plots of score histograms.
- Default:
True
- -v, --verbose¶
Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).
- Default:
0
- -H, --dump-config <dump_config>¶
Name of the config file to be generated
Arguments
- CONFIG¶
Optional argument(s)
Examples:
Evaluate the generated saliency maps for their localization performance:
mednet classify saliency interpretability -vv pasa tbx11k-v1-healthy-vs-atb --input-folder=parent-folder/saliencies/ --output-json=path/to/interpretability-scores.json
view¶
Generate heatmaps for input samples based on existing saliency maps.
It is possible to pass one or several Python
files (or names of mednet.config
entry points or module names) as
CONFIG arguments to the command line which contain the parameters listed below as Python variables. The options through the command-line (see below)
will override the values of configuration files. You can run this command with
<COMMAND> -H example_config.py
to create a template config file.
mednet classify saliency view [OPTIONS] [CONFIG]...
Options
- -m, --model <model>¶
Required A lightning module instance implementing the network to be used for applying the necessary data transformations.
- -d, --datamodule <datamodule>¶
Required A lightning DataModule containing the training, validation and test sets.
- -i, --input-folder <input_folder>¶
Required Path to the directory containing the saliency maps for a specific visualization type.
- -o, --output-folder <output_folder>¶
Required Directory in which to store the visualizations (created if does not exist)
- Default:
'visualizations'
- -G, --show-groundtruth, -g, --no-show-groundtruth¶
If set, visualizations for ground truth labels will be generated. Only works for datasets with bounding boxes.
- -t, --threshold <threshold>¶
Required The pixel values above
threshold
% of max value are kept in the original saliency map. Everything else is set to zero. The value proposed on [WWD+20] is 0.2. Use this value if unsure.- Default:
0.2
- -v, --verbose¶
Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).
- Default:
0
- -H, --dump-config <dump_config>¶
Name of the config file to be generated
Arguments
- CONFIG¶
Optional argument(s)
Examples:
Generate visualizations in the form of heatmaps from existing saliency maps for a dataset configuration:
mednet classify saliency view -vv pasa tbx11k-v1-healthy-vs-atb --input-folder=parent_folder/gradcam/ --output-folder=path/to/visualizations
config¶
Command for listing, describing and copying configuration resources.
mednet config [OPTIONS] COMMAND [ARGS]...
copy¶
Copy a specific configuration resource so it can be modified locally.
mednet config copy [OPTIONS] SOURCE DESTINATION
Options
- -v, --verbose¶
Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).
- Default:
0
Arguments
- SOURCE¶
Required argument
- DESTINATION¶
Required argument
Examples:
$ mednet config copy montgomery -vvv newdataset.py
describe¶
Describe a specific configuration file.
mednet config describe [OPTIONS] NAME...
Options
- -v, --verbose¶
Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).
- Default:
0
Arguments
- NAME¶
Required argument(s)
Examples:
mednet config describe montgomery
mednet config describe montgomery -v
list¶
List configuration files installed.
mednet config list [OPTIONS]
Options
- -v, --verbose¶
Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).
- Default:
0
Examples:
mednet config list
mednet config list -v
database¶
Command for listing and verifying databases installed.
mednet database [OPTIONS] COMMAND [ARGS]...
check¶
Check file access on a database configuration split.
mednet database check [OPTIONS] ENTRYPOINT
Options
- -l, --limit <limit>¶
Required Limit check to the first N samples in each split in the configuration, making the check sensibly faster. Set it to zero (default) to check everything.
- Default:
0
- -v, --verbose¶
Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).
- Default:
0
Arguments
- ENTRYPOINT¶
Required argument
Examples:
Check if all files from the config split ‘montgomery-f0’ of the Montgomery database can be loaded:
mednet database check -vv montgomery-f0
list¶
List all supported and configured databases.
mednet database list [OPTIONS]
Options
- -v, --verbose¶
Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).
- Default:
0
Examples:
$HOME/.config/mednet.toml
), and add a line like the following:[datadir] montgomery = "/path/to/montgomery/files"Note
This setting is case-sensitive.
$ mednet database list
preprocess¶
Run pre-processing on databases using a set of model transforms.
This command can be used to generate more compact versions of the databases that can be used to speed-up data loading. This is done by running both data loading and model transforms so that stored samples are small enough, and require no transformations when loading.
It is possible to pass one or several Python
files (or names of mednet.config
entry points or module names) as
CONFIG arguments to the command line which contain the parameters listed below as Python variables. The options through the command-line (see below)
will override the values of configuration files. You can run this command with
<COMMAND> -H example_config.py
to create a template config file.
mednet database preprocess [OPTIONS] [CONFIG]...
Options
- -o, --output-folder <output_folder>¶
Required Directory in which to store results (created if does not exist)
- Default:
'results'
- -m, --model <model>¶
Required A lightning module instance implementing the network to be trained
- -d, --datamodule <datamodule>¶
Required A lightning DataModule containing the training and validation sets.
- -l, --limit <limit>¶
Required Limit preprocessing to the first N samples in each split in the configuration, making the job sensibly faster. Set it to zero (default) to preprocess everything.
- Default:
0
- -g, --grayscale¶
For images to be grayscale, if set.
- Default:
False
- -v, --verbose¶
Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).
- Default:
0
- -H, --dump-config <dump_config>¶
Name of the config file to be generated
Arguments
- CONFIG¶
Optional argument(s)
Examples:
Pre-process the Montgomery (classification) dataset using model transforms from the pasa model.
mednet database preprocess -vv montgomery pasa
Pre-process the CXR8 (segmentation) dataset using model transforms from the lwnet model.
mednet database preprocess -vv cxr8 lwnet
detect¶
Object detection specialized commands.
mednet detect [OPTIONS] COMMAND [ARGS]...
evaluate¶
Evaluate predictions (from a model) on a classification task.
It is possible to pass one or several Python
files (or names of mednet.config
entry points or module names) as
CONFIG arguments to the command line which contain the parameters listed below as Python variables. The options through the command-line (see below)
will override the values of configuration files. You can run this command with
<COMMAND> -H example_config.py
to create a template config file.
mednet detect evaluate [OPTIONS] [CONFIG]...
Options
- -p, --predictions <predictions>¶
Required Filename in which predictions are currently stored
- -o, --output-folder <output_folder>¶
Required Directory in which to store results (created if does not exist)
- Default:
'results'
- -t, --iou-threshold <iou_threshold>¶
IOU threshold by which we consider successful object detection.
- -b, --binning <binning>¶
Required The binning algorithm to use for computing the bin widths and distribution for histograms. Choose from algorithms supported by
numpy.histogram()
, or a simple integer indicating the number of bins to have in the interval[0, 1]
.- Default:
'50'
- -P, --plot, --no-plot¶
Required If set, then also produces figures containing the plots of performance curves and score histograms.
- Default:
True
- -v, --verbose¶
Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).
- Default:
0
- -H, --dump-config <dump_config>¶
Name of the config file to be generated
Arguments
- CONFIG¶
Optional argument(s)
Examples:
Run evaluation on an existing prediction output:
mednet detect evaluate -vv --predictions=path/to/predictions.json
experiment¶
Run a complete experiment, from training, to prediction and evaluation.
Accepts all of command-line options from the
train
script.This script is just a wrapper around the individual scripts for training, running prediction, and evaluating. It organises the output in a preset way:
└─ <output-folder>/ # the generated model will be here ├── predictions.json # the prediction outputs └── evaluation.json # the evaluation outputsNote that complete experiments do not have options related to specific task functionality. Only the most generic set of options is available. To execute specific tasks, use the specialized command-line interfaces available in task-specific command groups.
It is possible to pass one or several Python
files (or names of mednet.config
entry points or module names) as
CONFIG arguments to the command line which contain the parameters listed below as Python variables. The options through the command-line (see below)
will override the values of configuration files. You can run this command with
<COMMAND> -H example_config.py
to create a template config file.
mednet experiment [OPTIONS] [CONFIG]...
Options
- -o, --output-folder <output_folder>¶
Required Directory in which to store results (created if does not exist)
- Default:
'results'
- -m, --model <model>¶
Required A lightning module instance implementing the network to be trained
- -d, --datamodule <datamodule>¶
Required A lightning DataModule containing the training and validation sets.
- -b, --batch-size <batch_size>¶
Required Number of samples in every batch (this parameter affects memory requirements for the network). If the number of samples in the batch is larger than the total number of samples available for training, this value is truncated. If this number is smaller, then batches of the specified size are created and fed to the network until there are no more new samples to feed (epoch is finished). If the total number of training samples is not a multiple of the batch-size, the last batch will be smaller than the first, unless –drop-incomplete-batch is set, in which case this batch is not used.
- Default:
1
- -a, --accumulate-grad-batches <accumulate_grad_batches>¶
Required Number of accumulations for backward propagation to accumulate “ gradients over k batches before stepping the optimizer. This parameter, used in conjunction with the batch-size, may be used to reduce the number of samples loaded in each iteration, to affect memory usage in exchange for processing time (more iterations). This is useful interesting when one is training on GPUs with a limited amount of onboard RAM. The default of 1 forces the whole batch to be processed at once. Otherwise the batch is multiplied by accumulate-grad-batches pieces, and gradients are accumulated to complete each training step.
- Default:
1
- -D, --drop-incomplete-batch, --no-drop-incomplete-batch¶
Required If set, the last batch in an epoch will be dropped if incomplete. If you set this option, you should also consider increasing the total number of epochs of training, as the total number of training steps may be reduced.
- Default:
False
- -e, --epochs <epochs>¶
Required Number of epochs (complete training set passes) to train for. If continuing from a saved checkpoint, ensure to provide a greater number of epochs than was saved in the checkpoint to be loaded.
- Default:
1000
- -p, --validation-period <validation_period>¶
Required Number of epochs after which validation happens. By default, we run validation after every training epoch (period=1). You can change this to make validation more sparse, by increasing the validation period. Notice that this affects checkpoint saving. While checkpoints are created after every training step (the last training step always triggers the overriding of latest checkpoint), and this process is independent of validation runs, evaluation of the ‘best’ model obtained so far based on those will be influenced by this setting.
- Default:
1
- -x, --device <device>¶
Required A string indicating the device to use (e.g. “cpu”, “cuda”, “mps”, “cuda:0”). Can be passed as a string, or as an environment variable (e.g. MEDNET_DEVICE=cpu or MEDNET_DEVICE=mps).
- Default:
'cpu'
- --cache-samples, --no-cache-samples¶
Required If set to True, loads the sample into memory, otherwise loads them at runtime.
- Default:
False
- -s, --seed <seed>¶
Seed to use for the random number generator
- Default:
42
- -P, --parallel <parallel>¶
Required Use multiprocessing for data loading: if set to -1 (default), disables multiprocessing data loading. Set to 0 to enable as many data loading instances as processing cores available in the system. Set to >= 1 to enable that many multiprocessing instances for data loading.
- Default:
-1
- -I, --monitoring-interval <monitoring_interval>¶
Required Time between checks for the use of resources during each training epoch, in seconds. An interval of 5 seconds, for example, will lead to CPU and GPU resources being probed every 5 seconds during each training epoch. Values registered in the training logs correspond to averages (or maxima) observed through possibly many probes in each epoch. Notice that setting a very small value may cause the probing process to become extremely busy, potentially biasing the overall perception of resource usage.
- Default:
5.0
- -A, --augmentations <augmentations>¶
Models that can be trained in this package are shipped without explicit data augmentations. This option allows you to define a list of data augmentations to use for training the selected model.
- -B, --balance-classes, -N, --no-balance-classes¶
Required If set, balances the loss term to take into consideration the occurence of each target in the training and validation splits.
- Default:
True
- -W, --initial-weights <initial_weights>¶
Path or URL to pretrained model file (.ckpt extension), corresponding to the architecture set with –model. Optionally, you may also pass a directory containing the result of a training session, in which case either the best (lowest validation) or latest model will be loaded. Note these weights will be overwritten when re-starting training from a checkpoint. This parameter should only be used as a set of initial weights (when starting a new training session).
- -v, --verbose¶
Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).
- Default:
0
- -H, --dump-config <dump_config>¶
Name of the config file to be generated
Arguments
- CONFIG¶
Optional argument(s)
Environment variables
- MEDNET_DEVICE
Provide a default for
--device
Examples:
$ mednet experiment -vv lwnet drive --epochs=2
info¶
Provide information about the current installation.
mednet info [OPTIONS]
Options
- -v, --verbose¶
Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).
- Default:
0
Examples:
Provide information about the current installation:
mednet info
predict¶
Run inference on input samples, using a pre-trained model.
It is possible to pass one or several Python
files (or names of mednet.config
entry points or module names) as
CONFIG arguments to the command line which contain the parameters listed below as Python variables. The options through the command-line (see below)
will override the values of configuration files. You can run this command with
<COMMAND> -H example_config.py
to create a template config file.
mednet predict [OPTIONS] [CONFIG]...
Options
- -o, --output-folder <output_folder>¶
Required Directory in which to save predictions (created if does not exist)
- Default:
'results'
- -m, --model <model>¶
Required A lightning module instance implementing the network architecture (not the weights, necessarily) to be used for prediction.
- -d, --datamodule <datamodule>¶
Required A lightning DataModule that will be asked for prediction data loaders. Typically, this includes all configured splits in a DataModule, however this is not a requirement. A DataModule that returns a single dataloader for prediction (wrapped in a dictionary) is acceptable.
- -b, --batch-size <batch_size>¶
Required Number of samples in every batch (this parameter affects memory requirements for the network).
- Default:
1
- -d, --device <device>¶
Required A string indicating the device to use (e.g. “cpu”, “cuda”, “mps”, “cuda:0”). Can be passed as a string, or as an environment variable (e.g. MEDNET_DEVICE=cpu or MEDNET_DEVICE=mps).
- Default:
'cpu'
- -w, --weight <weight>¶
Required Path or URL to pretrained model file (.ckpt extension), corresponding to the architecture set with –model. Optionally, you may also pass a directory containing the result of a training session, in which case either the best (lowest validation) or latest model will be loaded.
- -P, --parallel <parallel>¶
Required Use multiprocessing for data loading: if set to -1 (default), disables multiprocessing data loading. Set to 0 to enable as many data loading instances as processing cores available in the system. Set to >= 1 to enable that many multiprocessing instances for data loading.
- Default:
-1
- -v, --verbose¶
Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).
- Default:
0
- -H, --dump-config <dump_config>¶
Name of the config file to be generated
Arguments
- CONFIG¶
Optional argument(s)
Environment variables
- MEDNET_DEVICE
Provide a default for
--device
Examples:
Run prediction on an existing DataModule configuration:
mednet predict -vv lwnet drive --weight=path/to/model.ckpt --output-folder=path/to/predictions
Enable multi-processing data loading with 6 processes:
mednet predict -vv lwnet drive --parallel=6 --weight=path/to/model.ckpt --output-folder=path/to/predictions
segment¶
Image segmentation specialized commands.
mednet segment [OPTIONS] COMMAND [ARGS]...
dump-annotations¶
Dump annotations in a given folder, after pre-processing.
It is possible to pass one or several Python
files (or names of mednet.config
entry points or module names) as
CONFIG arguments to the command line which contain the parameters listed below as Python variables. The options through the command-line (see below)
will override the values of configuration files. You can run this command with
<COMMAND> -H example_config.py
to create a template config file.
mednet segment dump-annotations [OPTIONS] [CONFIG]...
Options
- -o, --output-folder <output_folder>¶
Required Directory in which to save predictions (created if does not exist)
- Default:
'predictions'
- -m, --model <model>¶
Required A lightning module instance that will be used to retrieve pre-processing transforms.
- -d, --datamodule <datamodule>¶
Required A lightning DataModule that will be asked for prediction data loaders. Typically, this includes all configured splits in a DataModule, however this is not a requirement. A DataModule that returns a single dataloader for prediction (wrapped in a dictionary) is acceptable.
- -v, --verbose¶
Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).
- Default:
0
- -H, --dump-config <dump_config>¶
Name of the config file to be generated
Arguments
- CONFIG¶
Optional argument(s)
Examples:
Dump annotations for a dataset after pre-processing on a particular directory:
mednet segment dump-annotations -vv lwnet drive-2nd --output-folder=path/to/annotations
evaluate¶
Evaluate predictions (from a model) on a segmentation task.
It is possible to pass one or several Python
files (or names of mednet.config
entry points or module names) as
CONFIG arguments to the command line which contain the parameters listed below as Python variables. The options through the command-line (see below)
will override the values of configuration files. You can run this command with
<COMMAND> -H example_config.py
to create a template config file.
mednet segment evaluate [OPTIONS] [CONFIG]...
Options
- -p, --predictions <predictions>¶
Required Path to the JSON file describing available predictions. The actual predictions are supposed to lie on the same folder.
- -o, --output-folder <output_folder>¶
Required Directory in which to store results (created if does not exist)
- Default:
'results'
- -t, --threshold <threshold>¶
This number is used to define positives and negatives from probability maps, and used to report metrics based on a threshold chosen a priori. It can be set to a floating-point value, or to the name of dataset split in
--predictions
.- Default:
'0.5'
- -m, --metric <metric>¶
Required If threshold is set to the name of a split in
--predictions
, then this parameter defines the metric function to be used to evaluate the threshold at which the metric reaches its maximum value. All other splits are evaluated with respect to this threshold.- Default:
'f1'
- Options:
precision | recall | specificity | accuracy | jaccard | f1
- -s, --steps <steps>¶
Required Number of steps for evaluating metrics on various splits. This value is used when drawing precision-recall plots, or when deciding the highest metric value on splits.
- Default:
100
- -a, --compare-annotator <compare_annotator>¶
Path to a JSON file as produced by the CLI
dump-annotations
, containing splits and sample lists with associated HDF5 files where we can find pre-processed annotation masks. These annotations will be compared with the target annotations on the main predictions. In this case, a row is added for each available split in the evaluation table.
- -P, --plot, --no-plot¶
Required If set, then also produces figures containing the plots of performance curves and score histograms.
- Default:
True
- -v, --verbose¶
Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).
- Default:
0
- -H, --dump-config <dump_config>¶
Name of the config file to be generated
Arguments
- CONFIG¶
Optional argument(s)
Examples:
$ mednet segment evaluate -vv --predictions=path/to/predictions.json --output-folder=path/to/results
Runs evaluation from existing predictions, incorporate a second set of annotations into the evaluation:
$ mednet segment evaluate -vv --predictions=path/to/predictions.json --compare-annotator=path/to/annotations.json --output-folder=path/to/results
view¶
Display predictions generated by a model.
It is possible to pass one or several Python
files (or names of mednet.config
entry points or module names) as
CONFIG arguments to the command line which contain the parameters listed below as Python variables. The options through the command-line (see below)
will override the values of configuration files. You can run this command with
<COMMAND> -H example_config.py
to create a template config file.
mednet segment view [OPTIONS] [CONFIG]...
Options
- -p, --predictions <predictions>¶
Required Path to the JSON file describing available predictions. The actual predictions are supposed to lie on the same folder.
- -o, --output-folder <output_folder>¶
Required Directory in which to store results (created if does not exist)
- -t, --threshold <threshold>¶
This number is used to define positives and negatives from probability maps, and used to report metrics based on a threshold chosen a priori. It can be set to a floating-point value, or to the name of dataset split in
--predictions
.- Default:
'0.5'
- -m, --metric <metric>¶
Required If threshold is set to the name of a split in
--predictions
, then this parameter defines the metric function to be used to evaluate the threshold at which the metric reaches its maximum value. All other splits are evaluated with respect to this threshold.- Default:
'f1'
- Options:
precision | recall | specificity | accuracy | jaccard | f1
- -s, --steps <steps>¶
Required Number of steps for evaluating metrics on various splits. This value is used only when evaluating thresholds on a datamodule split.
- Default:
100
- -e, --show-errors, -E, --no-show-errors¶
Required If set, then shows a colorized version of the segmentation map in which false-positives are marked in red, and false-negatives in green. True positives are always marked in white.
- Default:
False
- -a, --alpha <alpha>¶
Required Defines the transparency weighting between the original image and the predicted segmentation maps. A value of 1.0 makes the program output only segmentation maps. A value of 0.0 makes the program output only the processed image.
- Default:
0.6
- -v, --verbose¶
Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).
- Default:
0
- -H, --dump-config <dump_config>¶
Name of the config file to be generated
Arguments
- CONFIG¶
Optional argument(s)
Examples:
$ mednet segment view -vv --predictions=path/to/predictions.json --output-folder=path/to/results
Produces visualisations for the given (dumped) annotations:
$ mednet segment view -vv --predictions=path/to/annotations.json --output-folder=path/to/results
train¶
Train a model on a given datamodule (task-specific).
Training is performed for a configurable number of epochs, and generates checkpoints. Checkpoints are model files with a .ckpt extension that are used in subsequent tasks or from which training can be resumed.
It is possible to pass one or several Python
files (or names of mednet.config
entry points or module names) as
CONFIG arguments to the command line which contain the parameters listed below as Python variables. The options through the command-line (see below)
will override the values of configuration files. You can run this command with
<COMMAND> -H example_config.py
to create a template config file.
mednet train [OPTIONS] [CONFIG]...
Options
- -o, --output-folder <output_folder>¶
Required Directory in which to store results (created if does not exist)
- Default:
'results'
- -m, --model <model>¶
Required A lightning module instance implementing the network to be trained
- -d, --datamodule <datamodule>¶
Required A lightning DataModule containing the training and validation sets.
- -b, --batch-size <batch_size>¶
Required Number of samples in every batch (this parameter affects memory requirements for the network). If the number of samples in the batch is larger than the total number of samples available for training, this value is truncated. If this number is smaller, then batches of the specified size are created and fed to the network until there are no more new samples to feed (epoch is finished). If the total number of training samples is not a multiple of the batch-size, the last batch will be smaller than the first, unless –drop-incomplete-batch is set, in which case this batch is not used.
- Default:
1
- -a, --accumulate-grad-batches <accumulate_grad_batches>¶
Required Number of accumulations for backward propagation to accumulate “ gradients over k batches before stepping the optimizer. This parameter, used in conjunction with the batch-size, may be used to reduce the number of samples loaded in each iteration, to affect memory usage in exchange for processing time (more iterations). This is useful interesting when one is training on GPUs with a limited amount of onboard RAM. The default of 1 forces the whole batch to be processed at once. Otherwise the batch is multiplied by accumulate-grad-batches pieces, and gradients are accumulated to complete each training step.
- Default:
1
- -D, --drop-incomplete-batch, --no-drop-incomplete-batch¶
Required If set, the last batch in an epoch will be dropped if incomplete. If you set this option, you should also consider increasing the total number of epochs of training, as the total number of training steps may be reduced.
- Default:
False
- -e, --epochs <epochs>¶
Required Number of epochs (complete training set passes) to train for. If continuing from a saved checkpoint, ensure to provide a greater number of epochs than was saved in the checkpoint to be loaded.
- Default:
1000
- -p, --validation-period <validation_period>¶
Required Number of epochs after which validation happens. By default, we run validation after every training epoch (period=1). You can change this to make validation more sparse, by increasing the validation period. Notice that this affects checkpoint saving. While checkpoints are created after every training step (the last training step always triggers the overriding of latest checkpoint), and this process is independent of validation runs, evaluation of the ‘best’ model obtained so far based on those will be influenced by this setting.
- Default:
1
- -x, --device <device>¶
Required A string indicating the device to use (e.g. “cpu”, “cuda”, “mps”, “cuda:0”). Can be passed as a string, or as an environment variable (e.g. MEDNET_DEVICE=cpu or MEDNET_DEVICE=mps).
- Default:
'cpu'
- --cache-samples, --no-cache-samples¶
Required If set to True, loads the sample into memory, otherwise loads them at runtime.
- Default:
False
- -s, --seed <seed>¶
Seed to use for the random number generator
- Default:
42
- -P, --parallel <parallel>¶
Required Use multiprocessing for data loading: if set to -1 (default), disables multiprocessing data loading. Set to 0 to enable as many data loading instances as processing cores available in the system. Set to >= 1 to enable that many multiprocessing instances for data loading.
- Default:
-1
- -I, --monitoring-interval <monitoring_interval>¶
Required Time between checks for the use of resources during each training epoch, in seconds. An interval of 5 seconds, for example, will lead to CPU and GPU resources being probed every 5 seconds during each training epoch. Values registered in the training logs correspond to averages (or maxima) observed through possibly many probes in each epoch. Notice that setting a very small value may cause the probing process to become extremely busy, potentially biasing the overall perception of resource usage.
- Default:
5.0
- -A, --augmentations <augmentations>¶
Models that can be trained in this package are shipped without explicit data augmentations. This option allows you to define a list of data augmentations to use for training the selected model.
- -B, --balance-classes, -N, --no-balance-classes¶
Required If set, balances the loss term to take into consideration the occurence of each target in the training and validation splits.
- Default:
True
- -W, --initial-weights <initial_weights>¶
Path or URL to pretrained model file (.ckpt extension), corresponding to the architecture set with –model. Optionally, you may also pass a directory containing the result of a training session, in which case either the best (lowest validation) or latest model will be loaded. Note these weights will be overwritten when re-starting training from a checkpoint. This parameter should only be used as a set of initial weights (when starting a new training session).
- -v, --verbose¶
Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).
- Default:
0
- -H, --dump-config <dump_config>¶
Name of the config file to be generated
Arguments
- CONFIG¶
Optional argument(s)
Environment variables
- MEDNET_DEVICE
Provide a default for
--device
Examples:
Train a Pasa model with the montgomery dataset (classification task):
mednet train -vv pasa montgomery
Train a Little WNet model with the drive dataset (vessel segmentation task):
mednet train -vv lwnet drive
Train a Densenet model with the shenzhen dataset, while starting from a set of pre-trained weights:
mednet train -vv densenet shenzhen --initial-weights=path/to/checkpoint.ckpt
train-analysis¶
Create a plot for each metric in the training logs and saves them in a .pdf file.
mednet train-analysis [OPTIONS]
Options
- -l, --logdir <logdir>¶
Required Path to the directory containing the Tensorboard training logs
- -o, --output-folder <output_folder>¶
Required Directory in which to store results (created if does not exist)
- Default:
'results'
- -v, --verbose¶
Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).
- Default:
0
Examples:
mednet train-analysis -vv -l results/logs
upload¶
Upload results from a classification experiment folder to GitLab’s MLFlow server.
It is possible to pass one or several Python
files (or names of mednet.config
entry points or module names) as
CONFIG arguments to the command line which contain the parameters listed below as Python variables. The options through the command-line (see below)
will override the values of configuration files. You can run this command with
<COMMAND> -H example_config.py
to create a template config file.
mednet upload [OPTIONS] [CONFIG]...
Options
- -p, --project-path <project_path>¶
Required Path to the project where to upload model entries
- Default:
'medai/software/mednet'
- -e, --experiment-folder <experiment_folder>¶
Required Directory in which to upload results from
- Default:
'results'
- -n, --experiment-name <experiment_name>¶
Required A string indicating the experiment name (e.g. “exp-pasa-mc” or “exp-densenet-mc-ch”)
- -r, --run-name <run_name>¶
Required A string indicating the run name (e.g. “run-1”)
- -l, --upload-limit-mb <upload_limit_mb>¶
Required Maximim upload size in MB (set to 0 for no limit).
- Default:
10
- -v, --verbose¶
Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).
- Default:
0
- -H, --dump-config <dump_config>¶
Name of the config file to be generated
Arguments
- CONFIG¶
Optional argument(s)
Examples:
Upload an existing experiment result from a path it resides on (with a default experiment name as {model-name}_{database-name} and a default run name as {date-time}):
mednet upload --experiment-folder=/path/to/results
Upload an existing experiment result with an experiment name:
mednet upload --experiment-folder=/path/to/results --experiment-name=exp-pasa_mc
Upload an existing experiment result with a run name:
mednet upload --experiment-folder=/path/to/results --run-name=run-1
Upload an existing experiment result with defining a size limit of 20MB for each file:
mednet upload --experiment-folder=/path/to/results --upload-limit-mb=20
version¶
Provide version of the current installation.
mednet version [OPTIONS]
Options
- -v, --verbose¶
Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messages), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).
- Default:
0
Examples:
Provide versio of the current installation:
mednet version