Forecast Accuracy Metrics

 Forecast Accuracy Metrics



We will be discussing here about the most commonly used forecast accuracy metrics mentioned below:

  • Forecast bias
  • MAD
  • MAPE
  • MSE
  • RMSE
  • MPE
  1. Forecast Bias
  • This metric will help us in understanding whether the model has over estimated or under estimated. If the value is positive then its overestimation and if its negative then underestimation. 
  • In order to get bias as a % of sales, we have Forecast Bias %
         
                                                                  fig: relexsoumtions


                                  

  • If the value is more than 100 % then its an over forecast and if its less than 100% its under forecast.
  • It is important metric in demand forecasting as it will tell us the over or under supply at the central warehouse or distribution centers. It does not give information on quality of detailed level of forecast.
  • Target is to achieve a 1 or 100% and the number -/+ tells the deviation.


       2. Mean Absolute Deviation (MAD)

  • This metrics will tell how large the error is in the forecast on an average. 
  • It is not considered for comparison as this gives an average error. For eg, if an of error 500 units is large for a sales of 1,000 units but is marginal for a sales of 50,000 units. 

                                                                    fig: relexsoumtions

  • MAE is generally used to compare the result of different forecast model on same product/data and not on different.
  • MAE value goes from 0 to infinity and hence it will be difficult to interpret the result at times. So, we have MAPE value which will give result of deviation in percentage.


        3. Mean Absolute Percentage Error

  • This metric will tell how many percent your forecast is off from actual sales value (on average). This is most commonly used in demand forecasting. 



  • One drawback of this metric is that it is impossible to calculate MAPE when the sales is zero.
  • This is the Arithmetic average, which will give same weightage for each sales of a product and gives high error for slow movers.
  • There is another MAPE which is volume-weighted average, which gives more weight to high sales. Drawback for this average is that large error in case of slow movers can get unnoticed.
  • Target is to achieve 0% error.

    How to calculate MAPE or MAD (detailed or aggregated level)?

  • Detailed metric average calculation


  • Aggregated average calculation

We can see in both the cases the error value is very different and hence it will change the complete interpretation of the forecast. We use detailed metric average calculation for changes like store replenishment which is depended on each of the product MAPE or MAD. Aggregated average calculation is used for business decisions on an aggregated level like planning to take resources from the distribution centers.


     4. Mean Percent Error (MPE)

This metric is similar to MAPE but without absolute. This is used to give an additional feature of positive and negative deviation comparison.
fig: imgur


     5. Mean Square error (MSE)


                                                                           fig: vedexcel

  • MSE value is more than MAE due to squaring the error value (especially in case of outliers). 
  • In case of MSE, error increases in a quadratic fashion unlike MAE where it is proportional increase.
  • Predicted error is heavily penalized in this case.

        6. RMSE

  • RMSE is better than MSE as in case of RMSE, the error units and actual units matches unlike MSE. 
  • RMSE represents standard deviation of residuals hence tell how large the residual is dispersed.


                                                                           fig: analyticsvidhya






Reference:
  1. relexsolutions
  2. Youtube
  3. Wikipedia






Comments

Post a Comment

Popular posts from this blog

Euclidean Distance and Manhattan Distance

Image
  Euclidean and Manhattan Distance Euclidean Distance and Manhattan Distance are the two distances that we generally use to find distance between two data points. Use of this distance can be seen in algorithms like KNN, Kmean and Hierarchical clustering. We can check if there are any similarities between the two data points with the help of these distances. Euclidean distance The smallest possible distance between any two points is named as euclidean distance. This distance is calculated based on Pythagoras theorem as shown in the below figure. Source:  cdn-images-1.medium.com This is the case for 2-D data.  For higher dimensions we can add the additional dimension. This is case for any number of dimensions. Euclidean distance is used where we calculate the distance between any two points in a straight line irrespective of whether there are any other data points in between them. Ex. when we google to check distance between Cochin and Delhi in case of flight we get a straight point from
Read more

k-nearest neighbor

Image
 KNN The k-nearest neighbor algorithm is a supervised machine learning algorithm that can be used to solve both classification and regression problems. Classification usage Let us understand the algorithm using the below figure where we have two classes of data points ( A and B).  Source:  www.kdnuggets.com The first step is to determine the K value ( 'k' here is the number of nearest data points from the new data point). Let us take k = 3 as per the above figure. We can see that 2 out of 3 neighbors are from class B. So, in this case we go with the majority votes, i.e., the new data point will be classified as class B. We can use either Euclidean or Manhattan distance to get the nearest neighbor. Regression usage Below we can see the difference in the regression and classification usage. We will focus on the left figure for regression under this topic. Source:  www.jeremyjordan.me Similar to the classification problem, here also we need to come up with a k value. If the k val
Read more

K means clustering

Image
 K means clustering K-means  is a non-parametric method of clustering where we pre define the number of clusters(non-parametric means computation complexity depends in number of samples). It is an unsupervised machine learning technique. Source: miro.medium.com From the above figure, the algorithm finds the similarity between points and group them into clusters (green, blue and red in this case). Let us see what are the steps required and how the algorithm arrived at 3 clusters as above. Determine K value:  K value is basically how many centroids is needed for our data to form the best clusters. Here we can take any value for beginning.  Initialize K points (randomly) as cluster centers in the plane( K = 3 in this case) Find the distance between the points nearer to the centroids:  Here we can calculate the distance either using Euclidean and Manhattan Distance (Click here to know more about these distance). Based on the shortest distance from the centroids, clusters are formed. Sele
Read more