Learning Module

This is where the magic happens 🧙🪄

The Learning Module is constructed using the open-source Python library PyCaret. You can find valuable information about PyCaret at the following links:

• PyCaret Gitbook

• PyCaret ReadTheDocs

• PyCaret GitHub

Overview Videos

Learning Module - Scene Creation

Learning Module - Understanding Pipelines and Results

---

Create a Scene

To create a scene, follow these steps:

1. Click on the Learning Module Icon.
2. Click on the "Create scene" button.

3. Enter a name for the new scene.

The application will generate a folder that includes:

• Your scene (.medml file).

• A folder for your scene models.

• A folder for your scene notebooks.

breast_cancer_scene                -> Scene folder
    ├───models                     -> Folder where models are saved
    ├───notebooks                  -> Folder where notebooks are saved
    └───breast_cancer_scene.medml  -> Scene file

---

Module Overview

Double click on the .medml file to open the scene.

Empty scene

1. Available Nodes

Structure of a Node

The Add options button opens a panel where you can select additional options. These options are automatically generated from the online ReadTheDocs documentation of PyCaret.

The links we use to explain the PyCaret specific functions refer to the Classification section of the PyCaret documentation. However, please note that these functions are also present in other machine learning types within PyCaret. For instance, you can find them in the Regression documentation of PyCaret as well.

Node	Description	Input	Output
	This serves as the starting point for an experiment. You have two options here: MEDomicsLab Standard: This option takes files from a learning folder, which may represent the learning set from static .csv files obtained through the MEDprofiles process. The node automatically detects files that match the format and displays them in a dropdown selector. You then need to choose which column from your selected .csv file(s) should be the target for prediction. If you select more than one file, all the selected files should have the same target column. Custom File: You can choose any .csv file from the dropdown selector displaying all your .csv files from your workspace. Then, you need to choose which column from your selected .csv file should be the target for prediction. The available options for this node correspond to the PyCaret setup function options that are not directly related to data cleaning.	-	Dataset
	This node enables you to clean your dataset. The available options for this node correspond to the PyCaret setup function options that are directly related to data cleaning.	Dataset	Dataset
	This node allows you to select a model and its associated parameters. It corresponds to the estimator parameter of the PyCaret create_model function. The models are directly sourced from the scikit-learn package. In fact, if you are seeking documentation about a specific model parameter, you will find it in the scikit-learn documentation.	-	Model_config
	This node allows you to train a model. The available options for this node correspond to the PyCaret create_model function options (except the estimator parameter, which is defined through the Model node).	Model_config AND Dataset	Model
	This node allows you to train and evaluate the performance of all estimators available in the PyCaret model library using cross-validation. The available options for this node correspond to the PyCaret compare_models function options.	Dataset	Model(s)
	This node allows you to load a model from a file. It takes as input a model from the ones you saved in your scene, displayed in a dropdown selector. The available options for this node are the ones available in the PyCaret load_model function, except the model name, which is replaced by the selected file.	Dataset	Model
	This node allows you to optimize a model. It is different from the other nodes as, when clicking on it, it opens a subflow containing other nodes. It gathers the optimization functions of PyCaret.**	Model	Model
	This node allows you to analyze a model. It gathers the analysis and model explainability functions of PyCaret. For now, only the plot_model function is implemented in our application.	Model	-
	This node allows you to finalize a model. The available options for this node correspond to the PyCaret finalize_model function options (except the estimator parameter, which is passed as input to this node).	Model	Model
	This node allows you to save a model. The available options for this node correspond to the PyCaret save_model function options (except the estimator parameter, which is passed as input to this node).	Model	-

**Optimize Node

When clicking on an Optimize Node, it opens a subflow as shown below, allowing you to construct pipelines for optimization.

Optimize Node

The available nodes differ from those displayed in the original flow. You can find useful information on how to use them on this page of the PyCaret GitBook. For additional information about the available nodes, please refer to the PyCaret ReadTheDocs documentation:

• Tune Model documentation is available here.
• Ensemble Model documentation is available here.
• Blend Models documentation is available here.
• Stack Models documentation is available here.
• Calibrate Model documentation is available here.

The name of the node is where you can edit the name. The green blocks (I/O nodes) represent the input and output of the Optimize node. They are movable, and to connect them, simply intersect them with your nodes.

2. Results Button

This button is used to view the results of the experiment. It is disabled until you run an experiment. Once you have run an experiment, a .medmlres file is created in your scene folder, containing the generated results from the experiment. In fact, if you quit the app, your generated results will still be available the next time you open the app.

Once you click on the See results button, you activate the Result Mode. In this mode, you cannot manipulate the scene to build pipelines. The Result Mode only allows you to select the pipelines you want to observe the results from. To exit the Result Mode, simply press the See results button again, which becomes the Results mode on button while the Result Mode is activated.

Refer to the Results Panel section for more information about the results.

3. Utils Menu

This menu contains different functionalities that can be used to help you build your scene.

Element	Description
Machine Learning type dropdown	This dropdown allows you to select the type of machine learning you want for your experiment. When changing the type, all settings are reset.
Play	This button allows you to run the experiment. You can find additional information about running the experiment here.
Garbage bin	This button allows you to delete all nodes in the scene.
Save	This button allows you to save the scene.
Load	This button allows you to load a scene from a file.

4. Minimap

This minimap allows you to navigate the scene and visualize the nodes present in it.

5. Flow Utils

This menu contains various functionalities that interact with the flow section.

Element	Description
Plus Button	This button allows you to zoom in the flow section.
Minus Button	This button allows you to zoom out the flow section.
Square Button	This button allows you to fit the flow section in the view.
Lock Button	This button allows you to lock the flow section. When locked, you can't move the flow section.
Map Button	This button allows you to show/hide the minimap.

---

Example

1. Creation of your Pipeline

Drag and drop nodes from the menu to create your own pipeline(s). Here is an example of a simple classification pipeline that takes a dataset, trains a Gradient Boosting Classifier model, and then plots the resulting AUC (Area Under the Curve) plot.

Simple scene example

Parameters

2. Run the Experiment

You can execute the experiment by clicking on the Run button.

A progress bar will appear, and upon completion of the experiment:

• A Toast message will confirm that the results have been saved.

• A new file (.medmlres) will be generated in your scene folder, containing results related to your scene (you can't open the .medmlres file).

• The See results button will be enabled.

3. Results Panel

When you click the See Results button, you will activate the Result Mode.

Result Panel

As you can see, a panel opens at the bottom with various collapsible items:

Pipeline's Results Viewer

They are called the Pipeline's Results Viewer.

On the left, you can see all the nodes in the pipeline. You can click on them to view associated results.

On the right, there is a button that generates a notebook for the associated pipeline.

You can also filter which pipeline(s) you want to see in the results panel by checking the nodes in the flow section. Refer to the Pipeline Selection box section for more information.

3.1 Notebook Generation

The generated file is saved in the notebook folder of the associated scene. You can open it directly from the application by right-clicking and selecting "Open in..." and then choosing, for example, VSCode.

You need to run an experiment to access this functionality.

The default code editor is not implemented yet, but it's coming soon! 😉

Automatically Generated Notebook opened in VSCode

---

Multiple Pipelines

When constructing your scene, you can connect multiple nodes to each other, creating multiple pipelines!

1. Single VS Multiple Pipelines:

Single pipeline

Single Pipeline Example

Multiple pipelines

Multiple Pipelines Example

2. Pipelines Selection box

Each runnable node has a checkbox on top of the node in results mode. Checked nodes, especially those in green, will be displayed in the results panel.

Selection box

Independent selection box

The independent selection box (green ones) indicates that every pipeline goes through this node. Therefore, it will always be shown in the associated results.

Pipelines Selection Example

---

What PyCaret does ?

PyCaret primarily implements functions from the scikit-learn library.

1. Initialization

At the beginning of a Machine Learning pipeline, you initialize your data using PyCaret's setup function, corresponding to the Dataset and Clean nodes in our Learning Module. The setup function requires a dataset and the name of the target column. PyCaret then initializes elements for the pipeline.

1.1. Test Data

PyCaret divides your dataset into two parts: the training set and the test set (controlled by the test_data parameter in the PyCaret setup function). The training data is employed to train and optimize your machine learning model, while the test data is reserved for evaluating the created model. The split is conducted using the scikit-learn train_test_split function (useful explanations about this function can be found here).

*In the figure, "Full Dataset" refers to our Learning Dataset.

The random sampling step is executed with the aid of a random seed, and each split is linked to a specific seed. By default, PyCaret randomly assigns a seed at the start of each pipeline execution. To ensure the replication of the same experiment with a consistent split, you can set this parameter in PyCaret (using the session_id parameter in the Dataset node), as demonstrated in our experiments in the instructional video. This ensures that your test and train data will remain consistent across all executions.

Here, you also have the option to define the test data yourself and provide it to PyCaret. However, this capability is not currently available in our application when using the MEDomicsLab Standard format.

1.2. Folds

Then PyCaret will define folds on the train data to use for the Cross-Validation part (which will be executed using the Train or Compare Models box). The definition of the folds will also be done using a random seed, which you can define through the session_id parameter of PyCaret. By default, PyCaret uses the StratifiedKFold method from sickit-learn to define the folds. The stratified method ensures that each class from the target is represented equally across each fold.

2. Training

There are two functions related to training in PyCaret: compare_models (corresponding to our Compare Models box) and create_model (corresponding to our Train node).

2.1. Compare Models

The compare_models function is used to train all the available models from PyCaret on the initialized data from the setup function of PyCaret (our Dataset and Clean nodes). The resulting table displayed shows you the mean of the Cross-Validation results of all the folds for each model. For example, if we have five folds, for each model, we train the model five times, using a different fold as validation data at each iteration. Then, we apply the trained model to the validation fold and keep the resulting metrics to calculate the mean with the validation results of the four other iterations (purple data from the split in the image shown below).

The output of the compare_model function is the best model found using a specified metric (Accuracy by default, AUC as we specified in our instruction video). If we set the n_select parameter (as shown in our instruction video), we return the specified number of models from the top of the list.

2.2. Create Model

The create_model function takes initialized data as an entry and a model (that you can define through our Model node). It works exactly the same way as the compare_models function, except that we only test one model, and the results table shows the Cross-Validation results of each fold.

3. Analyzing

The analyses made using our Analyze node are showing the metrics resulting from our trained models on the test data defined at the initialization of the experiment.

4. Finalize

The finalize function in PyCaret (represented by the Finalize node in our app) trains the model one last time on the entire dataset, which includes both the training data and the test data, without changing its parameters.

PyCaret ROC (Receiver Operating Characteristic)/AUC (Area Under the Curve) plots

The AUC plots generated by the PyCaret library are derived from the YellowBrick Python package, which extends the scikit-learn API. By default, the plot displays multiple curves:

• The ROC curve per class for each class computed using the one-vs-rest method (meaning that the considered class is treated as the positive class and all other classes as the negative class).

• The micro-average curve, which is calculated by summing up all true positives and false positives across all classes.

• The macro-average curve, which takes the average of curves across all classes.

We acknowledge that these curves can be a bit confusing, especially with binary classification.

While using the YellowBrick package directly, we can set parameters to display only the classic ROC curve. However, we haven't found a way to set these parameters through our application with PyCaret yet. We are currently working on fixing this issue.