To Find the Right Forecasting Method, Try All of Them


Cassandra in Greek mythology was given the gift of foresight by Apollo in exchange for favours. She took his gift and didn't reciprocate which was a bad move. Apollo's curse was that she knew the future but no one would believe her when she told them. She knew things like the fall of Troy before it happened, destined to be powerless to stop it.

This is probably the best model of foresight because knowing any future with certainty is going to change it. I.e. If you knew how you were going to die, you would probably take steps to avoid it.

Forecasting and prediction are two very different activities. I can predict rain by seeing cows lying down. It is subjective with not much of a scientific basis. With forecasting we try to take a more scientific approach and try to explain why we mathematically think the future will behave the way it will. All the assumptions being laid out.

Implemented properly this is great for business. It can give you a better understanding of your environment as a result, help you plan better and more efficiently. Reducing costs due to efficiency, increasing profits because of adaptability to the environment.

A good forecast integrates both scientific as well as judgemental factors. The uniqueness of the particular situation means that each forecasting activity needs to be considered for its individual merits.

Some of the things to consider when creating a forecast are ... 1. Is it even worth forecasting. Some events are just so random that trying to put a model of behaviour over it is just futile. But it is still worth while looking at the data to understand this. Even if it means creating systems and processes to cater to this unpredictability.

  1. Do you have the data available? There are a range of really fancy models that can help you with forecasting, but having a clean data set to feed into these models are where the challenge is. Some things to consider.

    • Is the cost of collecting the data more that the benefit of better forecasting.
    • Is the level of granularity (the level of detail) adequate for the particular use case? For instance if you need to plan weekly but the data is only collected monthly.
    • The quality of the data. E.g. is being collected from multiple sources in a consistent manner.
  2. The level of sophistication of the model. Can you just look at averages or do you need something like a neural network for prediction. Is the additional complexity of the model justified?

I will try not to make this post as non technical as possible. All the code for this post can be found in my Github repository in an iPython notebook along with the images that have been generated via code.

I would like to engage with anyone interested in this and would really like to open up conversations regarding this topic.

Basically we will be going through different forecasting methods, discussing the merits of them for the particular requirement in hand. We will also discuss a framework to on how to judge how good a forecast is.

Some Definitions

Training and testing data. A good way of testing how well a forecasting method fits to your data is to test it against some of the actual data. Here we take some of the data we have collected and see how different it is from the prediction.

Root Mean Squared Error (RMSE). How good a model is can be tested by how far away all predictions are from the numbers they predicted. RMSE measures the squared differences between the prediction and the actual data. Since the differences are squared it means that positives don't cancel out negatives. The root of all the differences brings it back to the same units that are being considered. So you can say the model is x units wrong.

The Process

The process will be just graphically trying to show how the different forecasting methods fit into the requirement and then selecting the best methods.

But first ...

The Data

Tableau has a nice data set of Superstore sales data that can be downloaded here.

It is a nice data set with clean data which doesn't need much to be done to it. This will be very unusual in the real world.

Alt

There are about 10K line items and 20 columns. There are different product categories so it is inadvisable to lump them all together in case different categories exhibit different patterns of behaviour and looking at a specific product may capture the patterns better.

Alt

And this seems to be the case looking at the categories separately.

Alt

Furniture looks like a good contender to be analysed. This is because it looks like it has year on year growth as well as seasonality within the year. It becomes more apparent when it is put into neat monthly buckets.

Forecast Methods

Mean Forecast

Simply the average of historical values. This could work well with unpredictable data that doesn't have cyclical, seasonal or trend patterns, which is not the case with our furniture data and therefore this type of forecast performs quite badly.

For Root Mean Squared Error (RMSE) usually the smaller this number the better the model fits to the data.

Alt

Naive Forecast

A Naive forecast just gets the last actual value that was recorded and expects that this will continue... naively. It is good for forecasting what your mortgage payment will be next year if you know that you won't be moving.

Again compared to the actual values it looks like this method is too simple to capture what is happening with the data.

Alt

Linear Trend Based on all History

A linear trend captures some of the variation but is only slightly better than the other methods. It captures the year on year growth but none of the seasonality.

Alt

Linear Trend Based on Monthly Mean

The trend for the year is captured using each month's averages. Compared with the annual mean it captures the variation that happens within the year but not the year on year trend. It performs a lot better meaning there is more variation within the year that within the years.

Alt

Polynomial Trend Based on Monthly Mean

The same as above but trying to fit a curve to the data rather than a straight line. This is generally useful when there is no clear straight line fit. Like there is exponential changes to the data. Think of how Bitcoin grew or Amazon's share price. A straight line will not capture that very well. In this case the error in the curve is more than the error with a straight line. So in this case a straight line will be a better fit.

Alt

These are the more complex models. The calculations are more involved. There are more variables to think about because these also try to capture trend, seasonality and cyclical variations or changes that don't fit into seasons.

Seasonal Naive Method

Average out each month's data and assume this will be what the next period will be. Furniture sales has both a historical trend as well as an annual trend. Although this would capture the variation within the year, it doesn't capture the growth over the entire data set. Looking at the linear trend based on all history, there is a clear growth and averaging. There is a nice change in RMSE compared to the previous models too.

Alt

ARIMA

Auto Regressive Integrated Moving Average is a popular time series forecasting method, that captures both seasonality and trend. It looks like it is not doing any better than the Naive Seasonal method because it is throwing out less errors.

This brings up a point about selection criteria, where you should favour a simple model over a more complicated one.

Alt

Holt Winters Forecasting Method

Another forecasting method like ARIMA with a slightly different approach. This is the best option available for the reports as the error rate is the lowest using this one.

Alt

There are additional methods available to you but in the end you may pick a few methods and based on your understanding of the business and create a forecast with bands between which you think the actual numbers will lie.

Forecasting is also subjective, as there are elements of judgement that go into the forecast. A good model accommodates all useful approaches and is able to separate out the purely statistical from the judgemental. It gives the user a break down of why the model thinks the number is what it is.

What do you think? If you would like to get in touch and discuss this further, it would be great to hear from you.

References

Some of the resources I used to create this post are below. There are all really well written.

  1. Understanding ARIMA
  2. An end to end forecasting project
  3. The Dataset