Ensemble Technique 1 : Random Forest

 Random Forest





Ensemble technique is a technique where we combine multiple models and random forest is an ensemble technique with multiple decision trees. Random forest comes under bagging technique (also know as bootstrap aggregation).


Bagging Technique




Source: Wikipedia


Consider the above picture, where we have an original data, D(say). Similarly we have  multiple base learners or base models or classifiers , M1, M2,...Mn(say). Each models is provided with a sample of data (D') and the sampling of data for each model will be row sampling with replacement. Each model will get trained on that sample of data.
Suppose once the model is trained, test data is provided for validation of each model. As each model will come up with an output, based on voting classifier the majority of the output will be taken as final output for classification. For example, if from above figure, if majority of the classifier give blue as the output, we take blue as our final output. In this case, bootstrapping is basically generating multiple models from a single data and aggregation is combining the output(in this case taking the majority option).


Random Forest
Now we know what a bagging technique means, so random forest is a bagging technique with multiple decision tree models.  


Source: My rough notebook



Suppose we have a data 'D' with d number of records and n number of columns. In this case we have multiple decision tree as our base models (M1, M2, M3, ...Mn). Here for sampling of data(D') to each decision tree models we do both row sampling with replacement(RS) and feature sampling with replacement(FS) together. One need to keep in mind is if the original data set has D number of records and the sample has D' number of records then always D>D' and same in case of features (m > n, where number of column of original data and sample are m and n respectively). 
When we get the test data, each decision tree will give us an output. For aggregating we use voting classifier and this is called aggregation.

Let us analyse why to use a ransom forest instead of a single decision tree model.

Reason 1
Decision tree has 2 properties:
  1. Low bias: If decision tree is created to its complete depth then the model will get trained properly for training dataset. That is the training error will be less in this scenario. 
  2. High variance: Decision tree models are prone to give larger amount of error when test data is used for validation. This would result in an issue we call as overfitting.
So in case of random forest, as there are multiple decision trees with high variance. But when we combine all the model with maximum vote aggregation, the high variance will get converted to low variance. Each set of RS and FS will make the single base decision tree model expert with that sample and hence will lead to high variance. To convert high to low variance, we do aggregation as mentioned.


Reason 2
Suppose we make changes in the total number of records in the original data with 1000 resords(say). Eg, we make changes in 200 records(say). In this case, these new records will get evenly distributed on all the decision trees and hence there would not be much impact (in terms of accuracy) on the decision tree.


Note: In case of a regression problem, we take either the mean or median of the decision tree outputs as aggregation depending on the distribution of the data.


References:
  1. YouTube Channel - Krish Naik
  2. Wikipedia

Comments

Post a Comment

Popular posts from this blog

Euclidean Distance and Manhattan Distance

Image
  Euclidean and Manhattan Distance Euclidean Distance and Manhattan Distance are the two distances that we generally use to find distance between two data points. Use of this distance can be seen in algorithms like KNN, Kmean and Hierarchical clustering. We can check if there are any similarities between the two data points with the help of these distances. Euclidean distance The smallest possible distance between any two points is named as euclidean distance. This distance is calculated based on Pythagoras theorem as shown in the below figure. Source:  cdn-images-1.medium.com This is the case for 2-D data.  For higher dimensions we can add the additional dimension. This is case for any number of dimensions. Euclidean distance is used where we calculate the distance between any two points in a straight line irrespective of whether there are any other data points in between them. Ex. when we google to check distance between Cochin and Delhi in case of flight we get a straight point from
Read more

Ensemble Technique 2 : AdaBoost

Image
 AdaBoost AdaBoost or Adaptive Boosting is a boosting technique that is used as an Ensemble Method in Machine Learning. In this algorithm, the weights are re-assigned at each and every instances with higher weight for wrongly classified instances. The main motive behind this algorithm is to convert the weak learners to strong learners. Source:  miro.medium.com Boosting Technique As we can see from the above figure, suppose the original data is D1 and the trained classifier seen below is a base learner BL1(say). BL1 can be any model/algorithm provided with a sample of data to train. After training the model, when the original data is used to validate the model, the incorrectly classified records are further treated by moving them to a next base learner (BL2). Now the BL2 will be trained with those wrong classified records and further if they classify some more records incorrectly then again a new base learning is created (BL3) and this process continues till we specify some limits to th
Read more

Forecast Accuracy Metrics

Image
 Forecast Accuracy Metrics We will be discussing here about the most commonly used forecast accuracy metrics mentioned below: Forecast bias MAD MAPE MSE RMSE MPE Forecast Bias This metric will help us in understanding whether the model has over estimated or under estimated. If the value is positive then its overestimation and if its negative then underestimation.  In order to get bias as a % of sales, we have Forecast Bias %                                                                             fig:  relexsoumtions                                    If the value is more than 100 % then its an over forecast and if its less than 100% its under forecast. It is important metric in demand forecasting as it will tell us the over or under supply at the central warehouse or distribution centers. It does not give information on quality of detailed level of forecast. Target is to achieve a 1 or 100% and the number -/+ tells the deviation.        2. Mean Absolute Deviation (MAD) This metrics
Read more