Blast From The Past: Statistical Musings

While I am taking some time off I thought I may as well repost some of popular educational posts which many of you may not have seen or completely forgotten about. The first one is from early 2012 and covers volatility and kurtosis:

If you run the math you will find that on a daily basis – annually – the skew is close to normal, and but  KURT (i.e. kurtosis) is  large. That is due to compounding error of the VIX distribution and overlapping seasonality data. But nothing is black and white – I believe in conditional statistics and offsetting tail distribution. Common sense tells us that when the VIX is > 25 the market will exhibit a different distribution than when below – especially when you run it seasonally.

Let’s cover some of this as there is a ton of really good information in there. By kurtosis Volar refers to thin tail vs. fat tail distribution. For instance – on a weekly basis what are the percentages of times you get a +/- 0.5% move. What are the percentage of times you get a more violent 2% move? A thin tail market gives you a higher concentration of moves within a small percentage range, let’s say between 0% – 0.5%. A fat tail market gives you a more spread out profile where you see a lifting of the shoulder ranges (i.e. 1% – 2%) and a drop in the middle range (i.e. 0% – 1%). So now that we covered the basics let’s talk about the weather.

Yes, I am not kidding. Every morning you subs get a TICK TrendDay alert – that is your daily weather report based on statistical information captured via the TICK within the first 45 minutes of the NYSE session. Quite often the report nails it on the head and sometimes (like Thursday and Friday) it is completely off the mark. But it’s an attempt to gauge the ‘type of the market’ to expect for the day. Is it going to be a volatile day? Is it going to be a down or up day? When watching the Zero or your own indicators – should you trade divergences or is the short term trend so strong that they don’t matter and VWAP touches are nothing but buying (or selling) opportunities?

The work Volar has been putting together is purely statistical by nature and let me tell you that over 60% of institutional traders use nothing but statistical analysis for their trading. I think that needs to be repeated – a majority of the prop desks run by big institutional players do not care about indicators, golden crosses (or showers), technical analysis in general. If they do then it’s only a minor aspect of what drives their trading algos. It’s almost purely statistical mixed with some AI, genetic algos, HFT stuff, you name it.

For those quant boys ‘conditional statistics’ is everything – I call it the weather. You can express the daily weather in the markets in various ways via statistical conditions – maybe it’s a small number of parameters and maybe it’s hundreds. In the end the system will produce a set of odds for you and perhaps a trading range, entry points, exits, etc. It’s like figuring out if you’re going to take a drive to the mall or take the kids to the park for soccer practice – it all depends on the weather. Only retail rats use the very same algos (i.e. indicators and charts) every single day. Institutional traders have long figured out that what really matters are the ‘road conditions’ and they adjust their trading accordingly.

Which is why I keep telling you guys to pick your markets. The Zero for instance is a very simple tool, obviously geared toward retail traders. It kind of took over where the old ticker tape left off – instead of listening to the racket of the ticker the Zero gives us a feeling of participation and momentum of the trend. That is very valuable I think and it has kept us out of a lot of trouble in the past.

When Volar talks about the VIX being over or under 25 then he refers to essential market conditions that affect the way we should trade. I myself have figured that out when I started writing black box trading systems like Geronimo and evil.rat. Geronimo for instance does very well when the VIX is below 30 but if it’s above then it starts having issues and I actually disable it. It’s the opposite with evil.rat (which I haven’t run for a while now for obvious reasons) – it kicks ass especially during extended high volatility periods as for instance during 2008.

This is what I mean by picking your markets and I am considering extending my work on that end in collaboration with Volar. My vision would be to produce daily and more sophisticated weather reports based on seasonality, volatility, and other essential parameters.

Above graph shows us non seasonally adjusted distributions. By offsetting distribution i mean the holiday’s narrow tails overlap the fall fat tails. (now the VIX complicates this even more as it compounds the data). So things may look “Gaussian” when they are actually not- they are two different distributions. If you break out the months – the aug-oct period is far far from normal and highly skewed to the downside with large fat tails. Where as the “holiday” period is the opposite – it is narrow tailed and not skewed. So the holidays are also not normal, but narrow. I will also add that the high correlation among stocks also breaks the rule of normal distribution and portfolio theory- moreover, non constant distribution of both beta and correlation both imply that gaussian statistics have limits on how one can use them. Or bottom line on gaussian: only use conditional statistics to measure an expected outcome given 2-3 conditions; moreover, never use portfolio theory or VAR because the variables are ever dynamic and not constant.

Basically there are essentially two market conditions that take turns during the year – do you recall Volar’s seasonality charts? There are ‘easy’ times (e.g. the Santa Rally) and then there are more turbulent times. BTW, when I refer to ‘easy times’ then I mean more contained and uni-directional markets. A tough time would be a large correction followed by a fast short squeeze – can be extremely profitable but it’s not easy to trade. If you try then you most likely will have to abandon habits/strategies that work very well during ‘easy times’. There are also months when there is little edge to be had and you should just stay the heck out (i.e. summer IMNSHO).

When Volar talks about the ‘holidays’ then he refers to November until the end of December. Vacation starts after May and that may be a good time to dip into low volatility strategies. The fireworks (i.e. fat tail moves) usually happen in the August to October time frame – I’m sure you are aware of the concentration of market crashes and large corrections that have all occurred in fall.

As for volatility being non-uniform that is my whole point the overlapping non-uniformness makes much look normal or gaussian, when in fact there is 2-8 actual separate, underlying distributions. The other thing i will say is when you run conditions – e.g. if sma > sma_2 or something, the outcome of that particular trade may have normal distribution – regardless of actual market distribution. Which is why I run stats on outcome of conditions – because there is a reason for a normal distribution, not overlapping randomness. I mean if the market is 7% >100ema with the VIX<16 versus the market 15%< 100ema and VIX > 40 those markets will have utterly different expected outcomes- and for good reason. I will say if you are going to test the golden cross or ma’s (on a daily basis) you need to test the slope of the MA (eg. up or down) and then you need to test whether or not the market has made a new high or not. The stats will be much more clear.

Gaussian – if you have used Photoshop you may remember the Gaussian noise filter. It’s basically if you have a more randomness and there’s no clear pattern. Gaussian distribution also measures how many and what concentration of data points fall within a particular measured range – you usually get a bell shaped curve with dropping shoulders toward the 2nd and 3rd standard deviation.

Now what Volar is saying here is key – and in plain old English it means that the outcome of running your statistics depends on the type of market conditions you are looking at. I give you an example – you may find that a simple moving average cross makes you money for months on end but suddenly it starts breaking. What happened? Most likely it’s the market conditions that changes. Perhaps volatility dropped below a threshold, maybe the number of stocks above/below their 200-day SMA reached a certain threshold, or perhaps it’s a combination of factors. The trick here (and that’s the holy grail for the quant crowd) is to come up with a number of test conditions that allow you to find an edge for a particular ‘type’ of market.

It’s very tempting to go overboard and keep adding a boat load of test conditions – after all the more better, right? Well, wrong! I actually do have some experience with that from my time of writing software for a living. Back then we wrote a piece of code and then wrote a few small tests for it to make sure we weed out the majority of bugs early on. You can try to test for everything you can think of but usually the 80/20 rule applies – you catch 80% of the bugs with just some basic and tried test conditions. The same type of thinking applies to writing trading strategies – the root of all evil is premature optimization. Instead of adding more rules or conditions I always try to first figure out why a strategy does not work well during a particular time period. And often it’s the (market) weather that has a bigger effect than shifting/optimizing the strategy’s test parameters. Which means a particular strategy has its time to be active and its time to be silent.

If it snows outside then you’re probably not thinking of heading to the beach for a skinny dip. And if it’s 90 degrees and 80% humidity then soccer practice may not be high on your list and shopping for clothes in that air conditioned mall with your girlfriend may be much less of a painless endeavor. You need to approach the market with a similar perspective – when looking at the tape try to give the weather a bit of consideration. Try to remember which types of trades worked best in what market conditions and you are already light years ahead of the average retail retard.