Types of Ensemble Models

In our previous discussion, we explored an overview of ensemble models. Essentially, we train multiple models (called weak learners) and make final predictions by combining predictions from all the models. There are two key aspects we can control here:

• Training Data

  We can either use the same training data for all models or create different training samples for each model to train.

• Model Type

  We can choose to use the same type of models (e.g., all models being decision trees with different sets of hyperparameters) or opt for different model types. For example, one could be a decision tree, while others could be logistic regression, SVM, KNN, etc.

By varying data or model types, we can generate four different combinations, as illustrated in Diagram 1.

• Same Training Data and Same Types of Weak Learners

  This combination is not very useful, as training the same model on the same data leads to identical predictions. Consequently, combining these predictions does not provide any additional benefit.

• Same Training Data and Different Model Types

  This technique is known as stacking. Weak learners can be any machine learning model such as decision trees, SVM, logistic regression, or KNN. Details of stacking are discussed in the next section.

• Different Training Data and Same Model Types

  Bagging and boosting are techniques that operate under these settings. Random Forest is an example of bagging, whereas AdaBoost and Gradient Boosting are examples of boosting algorithms. We will discuss each algorithm in detail in upcoming sections.

• Different Training Data and Different Model Types

  This approach is generally not used in practice, but readers may experiment with these settings.

Diagram 1: Different ensemble models can be generated by varying training data or model types.

We will discuss the following three types of ensemble techniques, along with the types of problems each method addresses:

• Stacking
• Bagging
• Boosting