Before diving into ensemble models, let's first understand why they are needed. To do so, we will revisit the problems of overfitting (high variance) and underfitting (high bias).

Underfitting

Underfitting occurs when the model is not trained sufficiently. It results in high training and validation errors. This issue is known as a high bias, low variance problem. The left plot in Diagram 1 illustrates underfitting, where the decision boundary is overly simplistic and fails to capture patterns in the data, leading to high training error.

Overfitting

Overfitting happens when a model starts capturing micro-patterns in the data, including noise. This leads to a loss of generalization capability. The model performs well on training data but shows high error on validation and test data. This scenario represents a low bias, high variance problem. The right plot in Diagram 1 illustrates overfitting.

Diagram 1: Underfitting, Just Right, and Overfitting

Various methods exist to address overfitting and underfitting. Ensemble models also help mitigate these issues. Some ensemble techniques, such as Bagging, address high variance, while others, like Boosting, help reduce high bias. We will explore these techniques in detail later.

The fundamental idea behind ensemble models is to rely on multiple models rather than a single one. Each individual model is called a weak learner. By combining predictions from multiple models, ensemble techniques improve accuracy and robustness.

Diagram 2 provides an overview of ensemble models. Multiple models are trained on the dataset, and final predictions are made by aggregating their outputs.

Diagram 2: Overview of Ensemble Model

How Ensemble Models Make Final Predictions

• Classification Model

  The final prediction is determined using a majority voting technique. The class that receives the most votes from individual models is chosen as the final output.

• Regression Model

  The final prediction is obtained by averaging the predictions made by individual models.