Forecast Accuracy Metrics

 Forecast Accuracy Metrics



We will be discussing here about the most commonly used forecast accuracy metrics mentioned below:

  • Forecast bias
  • MAD
  • MAPE
  • MSE
  • RMSE
  • MPE
  1. Forecast Bias
  • This metric will help us in understanding whether the model has over estimated or under estimated. If the value is positive then its overestimation and if its negative then underestimation. 
  • In order to get bias as a % of sales, we have Forecast Bias %
         
                                                                  fig: relexsoumtions


                                  

  • If the value is more than 100 % then its an over forecast and if its less than 100% its under forecast.
  • It is important metric in demand forecasting as it will tell us the over or under supply at the central warehouse or distribution centers. It does not give information on quality of detailed level of forecast.
  • Target is to achieve a 1 or 100% and the number -/+ tells the deviation.


       2. Mean Absolute Deviation (MAD)

  • This metrics will tell how large the error is in the forecast on an average. 
  • It is not considered for comparison as this gives an average error. For eg, if an of error 500 units is large for a sales of 1,000 units but is marginal for a sales of 50,000 units. 

                                                                    fig: relexsoumtions

  • MAE is generally used to compare the result of different forecast model on same product/data and not on different.
  • MAE value goes from 0 to infinity and hence it will be difficult to interpret the result at times. So, we have MAPE value which will give result of deviation in percentage.


        3. Mean Absolute Percentage Error

  • This metric will tell how many percent your forecast is off from actual sales value (on average). This is most commonly used in demand forecasting. 



  • One drawback of this metric is that it is impossible to calculate MAPE when the sales is zero.
  • This is the Arithmetic average, which will give same weightage for each sales of a product and gives high error for slow movers.
  • There is another MAPE which is volume-weighted average, which gives more weight to high sales. Drawback for this average is that large error in case of slow movers can get unnoticed.
  • Target is to achieve 0% error.

    How to calculate MAPE or MAD (detailed or aggregated level)?

  • Detailed metric average calculation


  • Aggregated average calculation

We can see in both the cases the error value is very different and hence it will change the complete interpretation of the forecast. We use detailed metric average calculation for changes like store replenishment which is depended on each of the product MAPE or MAD. Aggregated average calculation is used for business decisions on an aggregated level like planning to take resources from the distribution centers.


     4. Mean Percent Error (MPE)

This metric is similar to MAPE but without absolute. This is used to give an additional feature of positive and negative deviation comparison.
fig: imgur


     5. Mean Square error (MSE)


                                                                           fig: vedexcel

  • MSE value is more than MAE due to squaring the error value (especially in case of outliers). 
  • In case of MSE, error increases in a quadratic fashion unlike MAE where it is proportional increase.
  • Predicted error is heavily penalized in this case.

        6. RMSE

  • RMSE is better than MSE as in case of RMSE, the error units and actual units matches unlike MSE. 
  • RMSE represents standard deviation of residuals hence tell how large the residual is dispersed.


                                                                           fig: analyticsvidhya






Reference:
  1. relexsolutions
  2. Youtube
  3. Wikipedia






Comments

Post a Comment

Popular posts from this blog

Euclidean Distance and Manhattan Distance

Image
  Euclidean and Manhattan Distance Euclidean Distance and Manhattan Distance are the two distances that we generally use to find distance between two data points. Use of this distance can be seen in algorithms like KNN, Kmean and Hierarchical clustering. We can check if there are any similarities between the two data points with the help of these distances. Euclidean distance The smallest possible distance between any two points is named as euclidean distance. This distance is calculated based on Pythagoras theorem as shown in the below figure. Source:  cdn-images-1.medium.com This is the case for 2-D data.  For higher dimensions we can add the additional dimension. This is case for any number of dimensions. Euclidean distance is used where we calculate the distance between any two points in a straight line irrespective of whether there are any other data points in between them. Ex. when we google to check distance between Cochin and Delhi in case of flight we get a straight point from
Read more

Ensemble Technique 2 : AdaBoost

Image
 AdaBoost AdaBoost or Adaptive Boosting is a boosting technique that is used as an Ensemble Method in Machine Learning. In this algorithm, the weights are re-assigned at each and every instances with higher weight for wrongly classified instances. The main motive behind this algorithm is to convert the weak learners to strong learners. Source:  miro.medium.com Boosting Technique As we can see from the above figure, suppose the original data is D1 and the trained classifier seen below is a base learner BL1(say). BL1 can be any model/algorithm provided with a sample of data to train. After training the model, when the original data is used to validate the model, the incorrectly classified records are further treated by moving them to a next base learner (BL2). Now the BL2 will be trained with those wrong classified records and further if they classify some more records incorrectly then again a new base learning is created (BL3) and this process continues till we specify some limits to th
Read more