Forecast accuracy (%)

December 16th, 2010 by Sjoerd

Regarding the measurement of the forecast accuracy I would like to ask a question. The Formula 1 below shows a common (or intuitive) suggestion and is het one I would use as well. However, in SAP Formula 2 is applied and that makes me wondering if there is a best one.

Formula 1 favours the situation of underforecasting, while Formula 2 favours overforecasting.

Formula 1. FC acc (%) = max{ 1 – |Demand-Forecast|/Demand, 0}

Formula 2. FC acc (%) = max{ 1 – |Demand-Forecast|/Forecast, 0}

Do any of you have any further experience or thoughts about the difference in performance?

Kind regards,

Sjoerd

7 Responses to “Forecast accuracy (%)”

  1. Dira says:

    How to calculate this forecast

    A company is use a level production strategy and is to produce at the rate of 1000 units per month.
    Sales are forecast as 800,900,1000,1000,and 800 over the six months under review.The opening inventory is 500 units. If all period have the same number of working days,contract a production plan and calculate the ending inventory for each month end.

  2. hq neo says:

    I Will have to return again when my course load lets up – however I am taking your Rss feed so i could read your internet site offline. Cheers.

  3. K Croteau says:

    Reasons for using actual as the denominator when calculating forecast accuracy. The most important reason to measure accuracy is as a tool for improvement by identifying the error or bias and taking steps towards correcting it. Also, implementing Best Practices

    Reasons for using actual as the denominator:

    1. When analyzing multiple models each calculates a “different” forecast value, thereby you need a constant or basis of comparison which is the actual orders.
    • Model A – Forecast 130, Actual 100, Accuracy 70% (actual as denominator), Accuracy 76.9% (forecast as denominator)
    • Model B – Forecast 70, Actual 100, Accuracy 70% (actual as denominator), Accuracy 57.1% (forecast as denominator)

    In the above example, one would incorrectly identify Model A to be the more accurate Forecast Model.

    2. It is typically human nature to over forecast “feel good affect”, this includes New Product forecasting where there is an inherent bias for a product to do well. Other reasons are to secure availability of product when the forecaster is not accountable for inventory which could be internal (sales), external (customer), or the worst scenario where each has included a buffer.

    3. Finally, the magnitude at which accuracy declines when forecast is used as the devisor vs actual as the devisor. With actuals as the devisor the slope or rate is the same.

    Best Practices in measuring Forecast Accuracy:

    • Forecast Accuracy measured by Gross Profit vs. Revenue or Units. Not all units are created equal, hence we use Gross Profit which is scale independent (Example attached).
    • Calculating the number of Products that meet Target Accuracy or creating a Histogram with ranges.
    • MAD as a measurement rather than MAPE. The MAPE calculation produces large Absolute% Errors when demand is small and also ignores zeros and understates forecast errors.
    • Accuracy measured by Item/Location.
    • Not all product groupings have the same type of historical order pattern. Hence, low volume and intermittent demand pose a far greater challenge than high volume steady product categories or that not all product grouping can be forecasted determined by the coefficient of determination and coefficient of variation.

  4. Sjoerd says:

    Dear Rohan and Mark,

    Thanks a lot for your interesting comments. After I applied the formulas to some examples as well, I am truly convinced of your conclusions.

    Just to visualize our discussion, I made a graph of the forecast error as % of demand versus the forecas accuracy. Then, we clearly see the over forecasting tendancy when dividing by forecast.

    You can find the graph at: http://dl.dropbox.com/u/3691989/Do%20you%20prefer%20over%20forecastin.pdf

  5. Sjoerd,

    Thanks for the question and the comment. As Rohan correctly asserts, using the Actual demand as the denominator controls for the forecast bias. Since the forecast variable is under the control of the forecaster, when in doubt, there will be a propensity to forecast high.

    More so, if forecasting reports into Sales or into commercial operations, then the tendency to bias high is higher. High forecast bias pushes up production and inventories ensuring higher service levels but also has lower impact on the forecast error. On the other hand, if commercial operations or sales is also held responsible for inventories, then this bias factor is somewhat mitigated.

    I will now address your comment on extra stock. If there is high demand uncertainty which may result in higher observed forecast error, then inventory management policies should take this into account. Remember the simple safety stock formula includes a factor for the forecast error.

    Safety stock = Service Level * SQRT(LT) * Forecast Error

    So if sales on the other hand, unilaterally increases the forecast to cover for safety stock, then you are doing a double whammy ==> Higher forecast and higher production and higher unsold inventories, but also a safety stock portion that includes the extent of the error.

    If you have persistent forecast bias, then you have to control for forecast bias in your inventory management policies through ascertaining a tracking signal and see if that is triggered. If so, then you have to methodically adjust for bias in your total error component that goes into safety stock.

    This is a comedy of errors actually ==> increasing the forecast to cover for safety stock and then the supply chain adjusting the production plan for forecast bias and netting out bias in the safety stock calculations.

    In essence, this begins the vicious cycle of demand and supply gaming each other.

    So I would conclude that this is NOT an internal management decision. If you want to calculate forecast error correctly, you SHOULD use the actual demand as the denominator.

  6. Sjoerd says:

    Thanks for your reply. I agree on your conclusion that from a supply perpective we should divide by Demand. However, we can imagine the sale perspective, which prefers to have some extra on stock to acertain delivery to their customer(s). Therefore I would conclude it is an internal management decission.

  7. Traditionally demand planning is considered as a part of sales management. And that is the reason accuracy measures are measured in such a way that sales folks can analyze their sales attainment by looking at ‘sales/forecast’ ratio. Like you said, in second formula, wherein we are dividing by forecast, will tend to overforecast. Now forecast being denominator here will cause accuracy to be higher in overforecasting situations.

    Now think about this, these forecasts are input to supply chain. Over forecasting will result in higher inventories. Hence one will have to decide about the denominator wherein it can control this forecasting bias. So to overcome this forecasting bias it is recommended to divide by demand values.

    If you consider demand planning as part of supply chain (which is a trend in recent years) then supply chain will be responsible for inventory management.
    To conclude on this topic I will say divide by demand, by doing this you can focus on forecast accuracy.

    I will recommend you to read recent discussion on our Demand Planning LLC LinkedIn group about “Where would be the best place to position Demand Planning in an organization”?

    Demand Planning Net LinkedIn group: http://www.linkedin.com/groupItem?view=&gid=1808515&type=member&item=36987144&qid=ed860a37-fdd0-4cb1-af19-a2b2eae797b2&goback=.gmp_1808515