We will discuss a number of interesting examples where the power of AUTOBOX shines through. The examples will be both Univariate and Multivariate where Univariate is a single series (historical sales forecasted) and Multivariate encompasses supporting or helping variables in the sense of regression.
Consider a case where we only have 4 data points with each one taken one hour apart. By using data at each minute we are able to increase our sample size to 240 from 4. We are not really increasing the number of samples, but the statistical calculation is done as if we have, and so the number of degrees of freedom for the significance test is falsely increased and a spurious conclusion is reached. This is one of primary causes of "spurious correlation". By taking observations at closer intervals we create a time series with higher and higher autocorrelation. Our job is to somehow adjust for this type of intra-relationship (within series) and its effects on test statistics.
Let’s assume that if you were to test the significance of the slope for 4 data points and arrive at the conclusion that there is not a statistically significant slope. However, if we use the 240 observations and run our test we will be inducing significance. This is due to the fact that the statistical test is based on the ratio of the parameter to its standard error. The standard error will be lower due to the artificial increase in sample size and we will be falsely concluding significance when there is none.
Most forecasting textbooks, attempting to explain or sell the concepts present simple examples which require simple explanations and simple tools. For example, consider the case of monthly Ice Cream Sales in Norway. Couldn’t be easier!
Textbooks then proceed to develop alternative approaches on an ad hoc basis to study such phenomena. The normal focus is to list possible models without a clue as to how the models inter-relate . Model selection is done on either an out-of-sample basis or some within sample statistic. The out-of-sample approach fails a test of objectivity because it hinges on the number of values you have decide to withhold.
Here is an example of the typical model selection process offered by simple textbooks or simple software.
Very little discussion , if any is made of the Family Tree of forecasting and how models relate to each other.
The three components of a forecasting model are ( in no particular sequence )
.History of the series of interest
.Causal Variables i.e. potentially supporting series
Following is a traditional and usually insufficient presentation
This can be and should be expanded to
We know return to our simple example.
Note that when the Ice Cream data is scrutinized by AUTOBOX it returns a fairly straightforward forecast.
but so could all packages and approaches ! Many of which are cheaper and easier to use (understand) than AUTOBOX.
But on the other hand , in the real world the practitioner is often faced with more demanding examples:
This disconnect between the simple and the complex causes ad hoc modeling procedures to self-destruct. AUTOBOX takes this relatively complicated problem and reduces it to the following…….
Employing pattern recognition to deduce a quarterly effect. Notice the 3 high and 1 low pattern?
The idea is that complex approaches or tools should deliver simple answers when simple answers are appropriate and complex answers when complex answers are appropriate.
How can the average be unusual? If a value is too close to the mean, can it be detected as unusual ? When is the expected value equal to the average ?
We have just observed the value 5 at time period 1996/9. What to do ! How to forecast ? How to detect an Inlier ?
The problem is that you can't catch an outlier without a model for your data. Without it how else would you know that an observation violated that model? In fact, the process of growing, understanding, finding and examining outliers must be iterative. This isn't a new thought. Bacon, writing in Novum Organum about 400 years ago said: "Errors of Nature, Sports and Monsters correct the understanding in regard to ordinary things, and reveal general forms. For whoever knows the ways of Nature will more easily notice her deviations; and, on the other hand, whoever knows her deviations will more accurately describe her ways."
Some analysts think that they can remove outliers based on abnormal residuals to a simple fitted model sometimes even "eye models". If the outlier is outside of a particular probability limit (95% or 99%), they then attempt to locate if there is something missing from model. If not, they remove the observation. This deletion or adjustment of the value so that there is no outlier effect is equivalent to augmenting the model with a 0/1 variable (dummy) where a 1 is used to denote the time point and 0's elsewhere. This manual adjustment is normally supported by visual or graphical analysis ... which as we will see below often fails. Additionally this approach begs the question of "inliers" whose effect is just as serious as "outliers”. Inliers are " too normal or too close to the mean" and if ignored will bias the identification of the model and its parameters. Consider the time series 1,9,1,9,1,9,1,9,5 and how a simple model might find nothing exceptional whereas a more rigorous approach model would focus the attention on the exceptional value of 5 at time period nine.
To evaluate each and every unusual value separately is inefficient and misses the point of intervention detection or data-scrubbing. A sequence of values may individually be within "bounds", but collectively might represent a level shift that may or may not be permanent. A sequence of "unusual" values may arise at a fixed interval representing the need for a seasonal pulse "scrubbing”. Individual values may be within bounds, but collectively they may be indicative of non-randomness. To complicate things a little bit more, there may be a local trend in the values. In summary there are four types of "unusual values" ;
1. Pulse, 2. Seasonal Pulse, 3. Level Shift and 4. Time Trends .
In order to assess an unusual value one needs to have a prediction. A prediction requires a model. Hopefully the model utilized is not too simple, just simple enough.
The evidenced pattern is exploited and used to identify the anomaly at the most recent point in time. The history of the series is
1,9,1,9,1,9,1,9,5 whoops !
Thus the required action is clearly evident from a visual examination. AUTOBOX duplicates the eye’s detection of an unusual value …..reports it and in this case ignores it.
Some answers ….the mean is equal to the expected value when the series or set of vales can be described by a Normal Distribution (uncorrelated, constant variance, no unusual vales, constant model over time).
When you try and shoehorn or force a model into a pre-defined set or list then funny (bad) things can happen. Consider a series of 27 monthly values.
One of our competitors, when analyzing this series, concludes that it has both trend and seasonal factors ….neither of which can be supported by a “t test” or an “F test”. They misconstrue a shift in the mean at the 14th data point to signal permanent upward trend and the four unusual values to represent statistically significant seasonal patterns.
AUTOBOX on the other hand detects both the level shift and the four unusual values and responds to the new mean or level and ignores or discounts the four anomalies as not being part of the underlying process and therefore should not be propagated into the future.
It is somewhat disconcerting to think that users of this competing software are gullible enough to accept such analysis even if it originates in North Carolina. It just proves the point that it is not sufficient to obtain the best parameter estimates one also needs the best model. Limiting the solution to a pre-defined set can be disastrous.
Before leaving this consider the details of the inappropriate solution:
Here the models from the ASSUMED “pick best” list of models:
More models from the list (Note: The full list of models is INIFINITE so anything less is less)
As positive thinkers we are all inclined to project upwards …..
Consider the following example
Does this series evident a trend? To some ….the answer is yes …. to AUTOBOX the answer is a resounding not proven.
Compare this to a series that has evidenced and statistically proven trend and in this case two distinctively different trends.
Some apparently simple problems often are not so simple. Consider daily telephone calls recorded by British Tel over a period of 55 days. Can you detect an important outlier or unusual value? I couldn’t until I was told by AUTOBOX where to look for it!
This is a very “busy” graph ….full of information and as we shall “see” some misinformation.
where Red indicates actual (dots) and Black (triangles) reflect the series cleansed of outliers. Note how our eye had missed the very unusual activity at time period 16 and 17.
This leads naturally to the following robust forecast where the model parameters are not affected by the anomalies.
Orson Bean or maybe it was so somebody else, once said “ A trend is a trend …until it bends and when the trend bends the trend is at an end“. Local time trends in time series require identification of the break points and then estimation of the local trend. Consider the following real time series from one of the largest manufacturer of cigarettes.
AUTOBOX detects and exploits the evidenced break point leading to
Given that one has observed or collected N data values, how does one determine how many of them to use in constructing a model. The answer to this question requires a study of the possible transient nature of model/coefficients over time. We show a time series that has “a lumpy demand structure“ for the first 50 months and then a “ random walk demand “ over the last 50 points. Note that the second half (observations 51 to 100) more closely follow an AR (1) model with value close to .9 .as compared to the -.9 value for the first 50 data points.
Notice the difference between the two regimes. AUTOBOX automatically scans and finds this break point and then uses the most recent set (proven to be homogenous) in order to construct the model and the forecast.
Some people get confused by programs that fit versus programs that model. Consider the following real world example of quarterly shipments.
Submitting this analysis to a fitting program that tries a sequence (limited number) with no feedback from the error structure in order to improve the model.
with visually obvious deficiencies
Here are the residuals from the process which are clearly inadequate.
We now compare the results of AUTOBOX
notice that “FITOLOGISTS” never forecast !
Autobox’s residuals showing no apparent deficiencies.
Here are the actuals and the residuals together:
AUTOBOX detected changes in trend and variance, leading to a sufficient model.
Consider the following regression result where we are evaluating the relationship between a variable representing a “before and after variable”.
The standard ordinary least squares regression test concludes that the cause variable is statistically significant. Really!!!
The reason for the significance is that the mean of Y for the first 30 observations is statistically significantly different from the mean of Y for the last thirty observations. You say , correctly ….who cares ! The reason being that we are not , in this case, concerned with a test of the hypothesis of two means because the series has trend.
As compared to AUTOBOX’s automatic model identification which reject the assertion that the candidate X aids the understanding or behavior of Y.
When you put on a promotion, you usually increase sales (for that period), but sometimes (often) you reduce sales in period(s) following the promotion (LAG effect). This is known as relocating demand. Following is a graph of the history of sales depicting the promotion points.
Notice the “morning after effect” where sales are reduced because of the prior periods increase.
AUTOBOX identifies the pattern and then projects the pattern into the future.
Sometimes after you pre-announce a promotion you lose sales prior to the promotion period. This is the case of a LEAD effect. Following is a graph depicting a typical lead effect.
Notice how the wise customer reduces his demand prior to the promotion period. The circular for your local grocery chain infoms you about next week’s sales.
AUTOBOX does and concludes
Consider the case where the regression effect between Y and X depends on the size of X. This happens quite naturally with the sales of heaters as it relates to temperature. For temperatures above 50 degrees there is no relationship while when the weather dips below 50 degrees sales increase, as it gets colder. Also, the sale of fans as it relates to weather (temperature). When the weather is under say 65 degrees there is no relationship while as it gets warmer (above 65 degrees) sales go up.
This phenomena occurs in the drug industry with sales of a drug might increase when the pollen count exceeds a critical value or in the utility industry where as it get warmer you need less electricity to heat the house and then as it gets warmer yet you need more electricity to cool your house.
This is demand for Y as it relates to X. Note how the regression changes. Clearly an average regression would be inadequate. One has to identify the critical value where the regression changes and then locally (in terms of X) develop the appropriate coefficients.
Some definitions first:
Spurious Relation (or Correlation) (a) - A situation in which measures of two or more variables are statistically related (they cover), but are not in fact causally linked—usually because the statistical relation is caused by a third variable. When the effects of the third variable are taken into account, the relationship between the first and second variable disappears.
Lurking Variable - A third variable that causes a correlation between two others - sometimes, like the troll under the bridge, an unpleasant surprise when discovered. A lurking variable is a source of a spurious correlation.
An investigator who takes their correlation results as indicating a causal relationship is subject to a plentiful source of criticisms - the artifact of the third variable. If it were asserted from a significant correlation of A with B that A causes B. The critic can usually rebut forcefully by proposing some variable C as the underlying causal agent for both A and B.
A side bar: In "Lies, Damn Lies & Statistics" (1933) M.S. Bartlett wrote a paper in the JRSS entitled "Why do we sometimes get nonsense correlations with time series?".
The following is a tongue in cheek example of how two economic variables have a cross-correlation, but when studied with AUTOBOX the predictor variable is not proven to have any information.
Using AUTOBOX we get:
This apparent cross-correlation is based on the (incorrect) assumption that the series have no internal autocorrelation. Once you account for the time series or auto-projective component the cross-correlation becomes non-significant. The concept of incorporating needed, but omitted variables is clear in time series because the concomitant variable is usually obvious after its presence has been detected via ARIMA structure with a conclusion that the X variable is not significant above and beyond the historical impact of Y. There are many cases where the ARIMA structure reduces the error variance enables a clearer picture and the significance of a candidate variable. This process of identification usually arises when conclusions are drawn like "fireman cause damage because the more fireman you have at a fire the more damage is reported” or "storks bring babies based upon a per-capita analysis" or "the number of words in a person's vocabulary depends on their foot size" or "beer is a snob good because actual price was used rather than real price". Another interesting question related to this is “did you ever notice that there seems to be a correlation between the number of churches and the number of bars in a city” …..perhaps church attendence causes drinking ?