Usage

This package supports a fully reproducible research experimentation cycle for medical image classification, segmentation, and object detection with support for the following activities:

  • Training: Images are fed to a deep neural network that is trained to match labels (classification), reconstruct (segmentation), or find objects (detections) automatically, via error back propagation. The objective of this phase is to produce a model. We support training on CPU and a few GPU architectures (cuda or mps).

  • Prediction (inference): The model is used to generate predictions

  • Evaluation: Predictions are used evaluate model performance against provided annotations, or visualize prediction results overlayed on the original raw images.

We provide command-line interfaces (CLI) that implement each of the functions above. The commands should be called in sequence to generate intermediate outputs required for subsequent commands:

# SPDX-FileCopyrightText: Copyright © 2024 Idiap Research Institute <contact@idiap.ch> # # SPDX-License-Identifier: GPL-3.0-or-later digraph cli { fontname = "Helvetica"; bgcolor = invis; node [shape = rectangle, style = filled, fontname = "Helvetica", fillcolor = gray90, color = black]; graph [style = dotted, color = black, fontcolor = black]; edge [color = black]; rankdir = LR; subgraph data { cluster = true; rank = same; label = "datamodule"; node [shape = cylinder, fillcolor = gold, fixed = true, width = 1.5, height = 0.7]; train_data [ label = "Training", href="../data-model.html", target="_top", ]; valid_data [ label = "Validation", href="../data-model.html", target="_top", ]; test_data [ label = "Test", href="../data-model.html", target="_top", ]; } subgraph core { cluster = true; label = "experiment (command)"; subgraph products { cluster = false; node [shape = note, fillcolor = aquamarine]; model [label = "Model\nweights", fillcolor = coral]; train_metadata [label = "JSON\nmetadata"]; logs [label = "Tensorboard\nlogs"]; train_analysis_plots [label = "Train\nEvolution\nPlots"]; prediction_metadata [label = "JSON\nmetadata"]; predictions [label = "Predictions"]; evaluation [label = "Evaluation\nTable&Plots"]; evaluation_metadata [label = "JSON\nmetadata"]; } subgraph commands { cluster = false; train [ label = "train", href="../cli.html#mednet-train", target="_top", ]; train_analysis [ label = "train-analysis", href="../cli.html#mednet-train-analysis", target="_top", ]; predict [ label = "predict" href="../cli.html#mednet-predict", target="_top", ]; evaluate [ label = "evaluate\n(task specific)" href="../cli.html#mednet", target="_top", ]; } subgraph workflow { cluster = false; {train_data valid_data} -> train -> {model train_metadata logs}; {train_metadata logs} -> train_analysis -> train_analysis_plots; {test_data model} -> predict -> {predictions prediction_metadata}; {predictions} -> evaluate -> {evaluation evaluation_metadata}; } } }

# SPDX-FileCopyrightText: Copyright © 2024 Idiap Research Institute <contact@idiap.ch> # # SPDX-License-Identifier: GPL-3.0-or-later digraph cli { fontname = "Helvetica"; bgcolor = invis; node [shape = rectangle, style = filled, fontname = "Helvetica", fillcolor = gray90, color = white]; graph [style = dotted, color = white, fontcolor = white]; edge [color = white]; rankdir = LR; subgraph data { cluster = true; rank = same; label = "datamodule"; node [shape = cylinder, fillcolor = gold, fixed = true, width = 1.5, height = 0.7]; train_data [ label = "Training", href="../data-model.html", target="_top", ]; valid_data [ label = "Validation", href="../data-model.html", target="_top", ]; test_data [ label = "Test", href="../data-model.html", target="_top", ]; } subgraph core { cluster = true; label = "experiment (command)"; subgraph products { cluster = false; node [shape = note, fillcolor = aquamarine]; model [label = "Model\nweights", fillcolor = coral]; train_metadata [label = "JSON\nmetadata"]; logs [label = "Tensorboard\nlogs"]; train_analysis_plots [label = "Train\nEvolution\nPlots"]; prediction_metadata [label = "JSON\nmetadata"]; predictions [label = "Predictions"]; evaluation [label = "Evaluation\nTable&Plots"]; evaluation_metadata [label = "JSON\nmetadata"]; } subgraph commands { cluster = false; train [ label = "train", href="../cli.html#mednet-train", target="_top", ]; train_analysis [ label = "train-analysis", href="../cli.html#mednet-train-analysis", target="_top", ]; predict [ label = "predict" href="../cli.html#mednet-predict", target="_top", ]; evaluate [ label = "evaluate" href="../cli.html#mednet", target="_top", ]; } subgraph workflow { cluster = false; {train_data valid_data} -> train -> {model train_metadata logs}; {train_metadata logs} -> train_analysis -> train_analysis_plots; {test_data model} -> predict -> {predictions prediction_metadata}; {predictions} -> evaluate -> {evaluation evaluation_metadata}; } } }

Fig. 1 Overview of core CLI commands for model training, inference and evaluation. Clicking on each item leads to the appropriate specific documentation. The workflow is the same across different task types (e.g. classification, segmentation or object detection), except for evaluation, that remains task-specific. The right implementation is chosen based on the type of datamodule being used.

The CLI interface is configurable using clapper’s extensible configuration framework. In essence, each command-line option may be provided as a command-line option, or as a variable with the same name, in a Python file. Each “configuration” file may combine any number of variables that are pertinent to a CLI application.

Tip

For reproducibility, we recommend you stick to configuration files when parameterizing the CLI applications. Notice some of the options in the CLI interface (e.g. --datamodule) cannot be passed via the actual command-line as it may require complex Python types that cannot be synthetized in a single input parameter.

Using our extensible configuration framework, we provide a number of Configuration files that can be used in one or more of the activities described in this section. Our command-line framework allows you to refer to these preset configuration files using special names (a.k.a. “resources”), that procure and load these for you automatically.

Commands