Training & Evaluation

Training the AI is a very important step in creating something useful from the data.

It can also be very challenging, require lots of compute time/energy and just when it looks like everything is coming together, the accuracy drops at the last second and the model doesn’t look much better than it did several days or weeks ago.

Therefore, it is important to establish good metrics and practices.

Topaz3 provides a training pipeline which performs k-fold cross-validation and records the input parameters, model architecture, training histories and results and stores copies of the finished models.

This allows you to concentrate on defining the best model without having to worry about boilerplate code to properly record the results.

Example

Here is the code for a simple neural network (which turned out to be quite effective) being trained through the training pipeline:

# https://github.com/DiamondLightSource/python-topaz3/blob/master/topaz3/training_models/example_cnn_basic.py
from typing import Tuple

from keras import Sequential, optimizers
from keras.layers import Conv2D, Dense, Dropout, Flatten, MaxPooling2D

from topaz3.training_models.training_pipeline import pipeline_from_command_line


def create_basic_cnn_model(input_shape: Tuple[int, int, int]):
    """Define the basic cnn model"""

    model = Sequential()

    model.add(
        Conv2D(16, kernel_size=(3, 3), activation="relu", input_shape=input_shape)
    )
    model.add(MaxPooling2D(pool_size=(2, 2)))

    model.add(Conv2D(64, kernel_size=(3, 3), activation="relu"))
    model.add(MaxPooling2D(pool_size=(2, 2)))

    model.add(Conv2D(128, kernel_size=(3, 3), activation="relu"))
    model.add(MaxPooling2D(pool_size=(2, 2)))

    model.add(Flatten())
    model.add(Dense(512, activation="relu"))
    model.add(Dropout(0.3))
    model.add(Dense(512, activation="relu"))
    model.add(Dropout(0.3))
    model.add(Dense(2, activation="softmax"))

    model.compile(
        loss="categorical_crossentropy",
        optimizer=optimizers.adam(lr=1e-5),
        metrics=["accuracy"],
    )

    return model


if __name__ == "__main__":

    pipeline_from_command_line(create_basic_cnn_model, rgb=False)

As you can see, the function create_basic_cnn_model returns a Keras model which can be defined as any other model, with an input parameter of the expected input shape as a 3-dimensional array.

This function is then passed to the pipeline_from_command_line function along with whether the model is designed to use a 3-channel RGB input or a single channel grayscale input.

See the following examples:

• Example Basic CNN
• Example Pretrained CNN

Underlying Training Functions

topaz3.training_models.training_pipeline.pipeline(create_model: Callable[[int, int, int], <sphinx.ext.autodoc.importer._MockObject object at 0x7ff94375deb8>], parameters_dict: dict)

Execute the pipeline on the model provided.

Reads all files in from the training directory path provided, gets their labels from the ai_labels table in the database provided.

Sets up Keras ImageDataGenerator for training images with scaling and extra parameters provided, and for validation images with scaling only.

Randomly mixes training data and creates k folds.

Trains on different fold for a run for the number of runs requested.

If test directory is provided, evaluates against test data and records that in evaluation folder.

Records in output directory the history and saves model for each run.

Parameters in parameters_dict:

• training_dir (required) - directory with training images
• database_file (required) - path to database with ai_labels table to get labels from
• output_dir (required) - directory to output files to (this name will be appended with date and time when the training was started)
• k_folds (required) - how many folds to create
• runs (required) - how many training runs to perform
• epochs (required) - how many epochs to use in each run
• batch_size (required) - size of batch when loading files during training (usually exact multiple of number of files)
• test_dir - directory with testing images
• slices_per_structure - how many images should be averaged into one structure during testing
• rgb - whether the model is expecting a 3 channel image
• image_augmentation_dict - dictionary of key-value pairs to pass as parameters to the Keras ImageGenerator for training images

Parameters:

• create_model – function which returns new Keras model to train and evaluate
• parameters_dict – dictionary of parameters for use in pipeline

topaz3.training_models.training_pipeline.pipeline_from_command_line(create_model: Callable[[int, int, int], <sphinx.ext.autodoc.importer._MockObject object at 0x7ff94375deb8>], rgb: bool = False)

Run the training pipeline from the command line with config file

Get parameters from the command line and pass them to training_pipeline in the parameter dict

Parameters:

• create_model – function which returns new Keras model to train and evaluate
• rgb – whether the model is expecting a 3 channel image

topaz3.training_models.training_pipeline.get_pipeline_parameters() → dict

Extract the parameters from a mixture of config file and command line using configargparse

Return parameters_dict:  dictionary containing all parameters necessary for pipeline