Education
Now Reading
Quantutorial – Standard Deviation
28

Quantutorial – Standard Deviation

by The MoleMay 11, 2017

Standard deviation. You see it mentioned all the time but if that inquisitive little niece of yours would ask you what it is, could you actually explain it? No, I’m not talking about googling the formula on your mobile and then telling her to scram and kick a ball or something. What I mean is explaining standard deviation (SD) in a way that actually makes sense and may lead to her to taking an interest in STEM sciences later in life. Yeah, I didn’t think so. And you call yourself a trader? Step aside and let uncle Mole handle this.

Alright Lucy (that’s her name), imagine you take your ball into the yard outside and start kicking it. Every time you kicked it you measure the distance. Warning: Since Lucy is a spoiled little American brat we’ll be talking about yards – which will cause European metric based children to experience instant cerebral hemorrhage. So you kicked it the first time and it went 10 yards. You write that number down and then kick it again. Unfortunately you tripped a little and so the ball only went 3 yards. It counts anyway so write it down. Next time you take a running start and the ball flies 20 yards. Excellent, that definitely gets recorded for posterity. Meanwhile your annoying little brother shows up and kicks the ball away from you. He’s half your size so it only went 6 yards but he kicks it again and scores 8. You run after the little bugger and kick it back about 12 yards. Who instantly starts bawling and just out of spite kicks that ball as far as he possibly can – 9 yards – through the kitchen window. Game over and of course you blame your brother for everything.

So let’s do the numbers – we have measured 7 kicks in yards:

2017-05-11_kicks

10, 3, 20, 6, 8, 12, 9

First we need to figure out the average or mean of the distances, which we get by adding all those numbers together and dividing it by 7. So that would be 68 divided by 7 which yields us 9.71. Let’s draw a line on our chart:

2017-05-11_kicks mean

Alright at this point it’s becoming clear that Lucy must have been feeding her ADD meds to her dog as she ran out to play about five minutes ago. Mental note to drop her from this year’s Christmas shopping list. So let’s explain the rest to that good-for-nothing brother of yours:

All we do from here is to deduct the mean (i.e. 9.71) from all the numbers:

0.29, -6.71, 10.29, -3.71, -1.71, 2.29, -0.71

Oh-ooooh – negative numbers. I don’t remember anyone kicking the ball backwards! What to do? Simple we square those numbers which gets us a positive series:

0.08, 45.08, 105.80, 13.80, 2.94, 5.22, 0.51

Now that’s weird. Seems like the negative numbers turned into very large exponents. Those are actually the relative ‘variances’ based on the mean and if we take the mean of all those numbers we get to… drum rolls… 24.78. That number represents the average variance from the mean.

So how about standard deviation? I got you covered. All you need to do is to square that number and you get to 4.987, which is the standard deviation. Finally!

Of course one really annoying thing about that imbecile of a brother is that he never trusted you. So he whips out his mobile phone and goes to an online SD calculator. ‘Ha-haaa!’ he shouts ‘you were wrong!! See here, the real standard deviation is 5.376. I knew you always sucked in math’.

Ooops that’s embarrassing, what happened? Well, tell your brother that he calculated the ‘sample SD’ while you gave him the ‘population SD’. What’s the difference? If you take only a sample from a larger population of numbers then you need to actually deduct 1 from the number samples when calculating your variance. Instead of dividing the sum of the squared values by 7 we divide them by 6.

2017-05-11_standard_dev_calc

Here’s a spreadsheet I put together so you don’t get confused. On the bottom left you see our ‘control number’ which Excel’s built-in SD function. So we are spot on!

2017-05-11_kicks_standard_dev

All that’s left to do for us now is to actually draw the SD value(s) to our chart, in this case a very simple bar graph. Just like with a Bollinger you take the mean and then add the SD of 5.37. For the negative SD you simply deduct the same SD. That gives us two lines shown above in green, one at 4.338 and another at 15.09 (for sample SD which is what everyone uses by default). If you count the numbers of bars (i.e. kicks) that fall within that standard deviation range you arrive at 5 out of 7 (the 2nd and 3rd are outside). That sounds about right as that is 71% within a very small sample of 7. Strangely when writing that little story for Lucy I just pulled those numbers out of my butt and it’s interesting that the numbers worked out so well. So there appears to be a natural and almost instinctive order to distribution patterns.

standard_deviation

Now if Lucy raided her mom’s cookie jar and due to an epic sugar rush kept kicking that ball for hours on end whilst collecting the distances odds suggest that the final tally would settle around 68% – a little over 2/3 of the sample size. Another way of phrasing it is that 68% of all the kicks would most likely fall within a standard deviation range of 4.338′ to 15.09′. And that is called ‘normal’ distribution or ‘Gaussian’ data.

But wait there is more. We can actually keep adding SD intervals simply by adding the same distance (i.e. 5.37) once again and then again. Which is visualized in the graph shown above – I’m sure you’ve come across it in the past. In this specific case the kicking range spanning 2 standard deviations would be between 0.87′ to 18.558′ and encompass over 95% of all the samples. Each standard deviation interval is also often referred to as ‘confidence intervals’ and that is what the nerds mean by ‘sigma’.

2017-05-11_sigmas

Here’s a handy table that shows you each confidence interval, assuming normal distribution (a topic we have covered in the past but will revisit in the near future). If you click on the table it’ll get you to the page where you can play with the numbers or add your own.

Six Sigma Event?

Remember last time when you read some article over on ZeroEdge suggesting the possibility of a ‘six sigma’ event? Well, if you count the rows in that table above then you realize that this puts us in the top 0.001% of all trading days, which is 1/1000th of one percent, or one out of 100,000. Wait a minute. 100,000 / 365 comes out to 273 years? Almost three centuries? But we had several major market crashes in the past 100 years alone, not even counting recent flash crashes:

  1. Florida Real Estate Craze in 1926
  2. The Great Depression starting in 1929
  3. The Big Crash of 1987
  4. The Asian Crisis starting in 1989
  5. The Dotcom Crash in 2000
  6. The Housing Bubble Crash in 2007.

That comes out to at least 6 large ‘unforeseen’ market events in the past century. If you would average that out over 300 years that would be 24 out of 100,000 trading days, or 0.024%. Deduct that from 100 and you get to 99.976. For it to be a six sigma event it would have to be > 99.999 but it’s smaller. It’s not even > 99.99% which would have made it a five sigma event. It is however > 99.9% so perhaps they should call them Four Sigma events instead?

What Did We Learn Today?
  1. Lucy loves kicking balls but apparently doesn’t like math very much. Then again she’s nine years old, what’s your excuse?
  2. Mole is a horrible uncle who tortures innocent children with math riddles (I charge per hour by the way).
  3. Standard deviation is a measure of how spread out numbers are.
  4. SD is the square root of the variance throughout all the samples.
  5. I should never listen to ZeroEdge, read Evil Speculator instead.
  6. Apparently financial risk is not being assessed correctly. Big surprise there!
  7. You should definitely buy Nassim Taleb’s new book (no I don’t earn any commissions from that link).

About The Author
The Mole
Mole created Evil Speculator amidst the chaos of the financial crisis in early August of 2008. His vision for Evil Speculator is a refuge of reason, hands-on trading knowledge, and inspiration for traders of all ages and stripes. You can follow him and his nefarious schemes at various social media waterholes below.
  • http://www.captainboom.com/ captainboom

    Zero needs a kickstart Boss.

  • Trouzzer_Snake

    What an awesome post, thank you! This obviously took some effort and appreciate it.

  • http://evilspeculator.com Sir Mole III

    Really? I just turned it on. On it!

  • http://evilspeculator.com Sir Mole III

    It’s in preparation for a very juicy guest post which requires a basic understanding of standard deviation which in turn leads into volatility models.

  • http://evilspeculator.com Sir Mole III
  • Trouzzer_Snake

    And here I was thinking you were just inspired… :)

  • Yoda

    Now that was hilarious. XD

  • http://evilspeculator.com Sir Mole III

    Two people wrote me about Zero problems but it’s working fine for me (via the same page). Anyone else having problems?

  • Trouzzer_Snake

    Working for me.

  • BTrader

    time was working but chart was frozen at 2395 then it started working after email was sent…thx!!

  • http://gerb-reloaded.blogspot.com/ Gold_Gerb

    Drive-by posting

    Bitcoin > $1800.

    If your Sigma event is too big, your model is probably broken.
    Excellent post. back to the pool!
    -GG

  • HD

    FWIW- Near perfect elliot wave off the 4/19 low. 5=1 2406, sellers came in, broke the 2-4TL, quantifying the wave >90% probability of retrace. Previous 4th= first target. Done! Notable fibonacci relationships the entire rally. What I refer to as Fibcontrol™ If you were following along on TWTR actually had the 2404 SPX from the double bottom last week for first entry. 40 point round trip based purely on pattern and elliot wave. No indicators. IF volatility returns efficacy for EW will only increase. https://uploads.disquscdn.com/images/1059f31699cac2953407aad82cac0c7f08fd425a259501338db0950e30034759.png https://uploads.disquscdn.com/images/cfc6e91f1733873defc92275f90c889e1646d4032fa5f9de542c7e32779d94a6.png

  • http://evilspeculator.com Sir Mole III

    That’s a relief…

  • http://www.captainboom.com/ captainboom

    Odd. We were 4 minutes into the trading day, and I wasn’t seeing any Zero updates. Time was updating, but not the chart. Working for me now.

  • BobbyLow

    Well done! :)

    And I think the lower the SD on System Results, the less chance of blowing up?

  • Yoda

    Oh look a sell-off.

  • http://evilspeculator.com Sir Mole III

    Oh, I just grasped what you said, sorry. Yes, a lower standard deviation usually means that your winners and losers are better distributed and thus will be psychologically easier to trade. Mean reversion systems usually have a lower SD than trend trading systems which have a low win/loss ratio and where the winners far exceed 1R or 2R on average.

  • http://evilspeculator.com Sir Mole III

    Must have been a TOS problem then. Unfortunately that is out of my control.

  • Yoda

    Nutty gas is looking nicely bullish today. Bobby, are you still in L UNG?
    https://uploads.disquscdn.com/images/d11e5750fbdf71de442173dbbb56469147797264ee1ffd98687abd10994a7027.png

  • BobbyLow

    Nope. I was trading Nutty Gas to see if it would be a good fit for me. I had mixed results with it. I didn’t have enough patience to trade the daily and trading NG with the hourly kind of sucked with my system so I put that horse back into the barn.

  • http://evilspeculator.com Sir Mole III
  • Mary

    Mr. Mole. We just arrived at your inflection point again that you mentioned 2 or 3 weeks ago.

  • http://evilspeculator.com Sir Mole III

    Still not going anywhere though – grrr…

  • CandleStickEmUpper

    Great post Mole. You have an excellent, clear way of explaining math. Ever think of writing an advanced college math book?

  • http://evilspeculator.com Sir Mole III

    I’d love to but I’m really bad at math 😉

  • Yoda

    Seasonality is supportive of a bull move until mid-June.
    http://charts.equityclock.com/natural-gas-futures-ng-seasonal-chart

  • Yoda
  • http://evilspeculator.com Sir Mole III