System Trading
Now Reading
Exercises in Trading System Building – Part 1
75

Exercises in Trading System Building – Part 1

by ScottMarch 8, 2017

This is Scott, I’m taking over from Mole while he gets some downtime in Tenerife this week.

I’d like to digress a little from the usual bullshit about markets (which of course I will cover), and work through the preliminary stages of working up a new system, from concept to backtesting, forward testing and optimisation. I’d also like to update you on some of the things I’ve learned since I last did a system building course for your guys in early 2014.

I’m going to have to eat a little humble pie along the way. Turns out a few of my deeply held market beliefs needed to be thrown away, and some things I argued strongly for were completely wrong. Both Mole and I have learned a lot since 2014, wasted months on some intractable problems, solved a few more, and discovered some fascinating ways to shoehorn objectively average systems into something much, much better. I’ve had exposure to real institutional traders who are orders of magnitude better than me, and had the opportunity to peek over their shoulders and steal their secrets like the rat cunning bottom feeding hustler I am.

I’ve also done some work investigating the systems of other pros and I have some insights into both trend following futures systems and stock trading mean reversion systems that are rich veins of knowledge to explore. I’ve gone as far as I can without having programming skills of my own so I’m forcing myself to start programming at age 42. I suck at that, and I most likely always will, but nothing worth doing is easy, so I persist. My plan is to use the superb software at quantopian as a research environment.

These are all works in progress for my own professional development, however, and I prefer not to write authoritatively on them until I am an expert and not just a dabbler.

If you were not around here back in the day, it might help you to take a refresher course

http://evilspeculator.com/arisen/

http://evilspeculator.com/building-your-trading-system-a-practical-guide/

http://evilspeculator.com/building-your-trading-system-part-3-choosing-your-market-type-and-finding-your-edge/

http://evilspeculator.com/building-trading-systems-part-4-entry-and-exit-techniques-and-system-optimization/

Without further ado, let’s dive right in. What I’d like to do is show you how the real pros work up a system. And truthfully it’s not a lot like my original approach, which had a tendency to build systems with too many moving parts. Too many moving parts means by definition FRAGILE. We are going to keep coming back to this concept of fragility, and for an interesting diversion on the concept I can recommend this book by Taleb (the black swan guy).

What the pros do is start with a hypothesis to test. They test that hypothesis to an acceptable degree of statistical significance BEFORE they start working up a system. Statistical significance is a pointy headed way of saying that small amounts of data don’t mean shit (which means that seasonality data in particular is suspect)

The old idea of working through market beliefs and building up a list of indicators and whatnot you believe in, has some merit for purely discretionary systems. Where I personally fucked it up was by taking a full system, and backtesting that, rather than the raw concept first. Also, it doesn’t really matter much what your beliefs are, if they aren’t an edge you are quite simply wasting your time.

Of course we don’t want to objectively test our own long held ideas about technical analysis. What if we are wrong? What if we have to admit that we wasted years of work on things that don’t have any validity at all? Well welcome to the club, I’ve wasted years of my life on garbage too. Suck it up snowflake and get testing, or scuttle off back to your safe space.

Lets look at one of the components of the Crazy Ivan system which was the first one Mole and I built. The inside period, going long on break of the high or short on break of the low with a stop a tick below the low (for a long) or a tick above the high (for a short). Let’s use a simplified 1.5R target to make things easy

inside period

Now, you can see in this instance we went long on break of the high, and it hit the 1.5R target for a nice win.

My old approach was to backtest this a LOT, but there is a big problem with that. I’m not just testing the hypothesis that a break of an inside period extreme leads to a move, I’m also testing the efficacy of my stop, my target, how quickly I move to breakeven, and any other indicators I might be adding. Each extra thing I’m adding adds an order of magnitude of fragility to the whole bag of cats.

So I can generate a 1000 trade backtest, and not even be sure if I’m looking at a coin flip edge with a good exit strategy, or a strong edge with a shitty exit, or anything in between. There are ways I can mitigate this, but bottom line, if I walked into the office at RenTech with this strategy they would laugh me out the door. Now think what happens when I mix 5 OTHER setups in with the already existing bag of cats. I have no idea what’s working and what’s not, whether the same exit and stop logic is optimal (or even decent) for the other setups, etc etc. It is a child’s bedtime story, good for making me feel better but completely and utterly useless in the real world.

My bad. I wasted a decade of my life on rubbish like this. I’m sorry if I led you astray, but hopefully I can lead you back to the path of righteousness. And for the record, Crazy Ivan is a small but measurable edge, just parts of it suck and the methodology to build it was wrong.

The upshot of it is that while I was stuck in this mode of building systems I could produce all the backtests I wanted, and they don’t mean a fucking thing. This has led Mole and I to be on the verge of throwing away backtesting completely (which is what Van Tharp did) which is also wrong. Backtesting is a difficult beast with many inherent problems that need to be overcome or mitigated, but just because it is difficult doesn’t mean we shouldn’t do it.

Let me introduce you to the gold standard for testing all your stupid trading ideas….The Scatter Plot (click the links) Second link

scatter plot

So what we do is test the broad concept of our trading ideas, one at a time. We plot some data points and see if we can roughly draw a line of best fit.  We are looking for a positive correlation for some things, and a negative correlation for mean reversion (obviously)

Some examples will illustrate. There is a very weak positive correlation with investor optimism and one month change in S&P. The technical term for how well the dots make a line is the “R squared” statistic (labelled on most of the charts) here you can see the R2 number of .06 is basically very weak line of best fit.scatter12 scatter1

potholesVsCrimeTotals_withline

earnings-growth-percent-scatter-plot-2009-to-2020-chart-equity-valuations

This is all covered well in Victor Neiderhoffer’s seminal and very well priced book which I recommend. How could we use this in a practical sense? Let’s say we wanted to test for mean reversion on SPY on a 2 day period. We plot the 2 day return on the X axis, against the return for the NEXT 2 DAYS on the Y axis. We have a reasonable -.17 negative correlation, which means that there is a legitimate statistical tendency for 2 up days in the SPY to be followed by 2 down days (measured as a total, not both days down). As an aside this tendency is also present (albeit to a lesser degree) in the Australian, Russian and Brazil equivalents, giving more confidence in the concept.

spy

This is really important, and its important to understand WHY this is better than just building a mean reversion system and backtesting it. If we build the system, it might test 100 trades, but in the course of those 100 trades it actually takes, it probably has thousands of events. We get more data points, more statistical significance, and purer data. Here we are JUST testing the tendency of price to mean revert on a 2 day period, as opposed to testing a system with a stop, take profit, trailing stop logic, and all the other stuff that goes into system building.

If we don’t start with an idea that is real, we can literally throw the backtest away, because we have built our house out of straw. The good news about scatter plots is they can be done by hand, in google sheets, excel. The other good news is they are far more reliable than backtests, because you are testing just one idea, and not the fragile bag of cats that is the entire system.

It also means that old idea about it doesn’t matter what instrument we trade or what timeframe we trade it all tastes like chicken and a chart is a chart is a chart – well that one can be tested. And empirical testing reveals I was completely wrong about that and BobbyLow you were completely right! Mea Culpa my friend. In my travels there are many markets which are empirically trending intraday and mean reverting interday (like ES futures for example). You just can’t take a system which works on one timeframe and throw it up on 5 min charts or 360m charts. Sorry, this is one of the things I really thought was true, but it’s not. Also, the most robust edges are present across markets, but the most robust edges are not the best edges. Not everything is present everywhere. Some markets gap and revert, some don’t. Currency markets have very few mean reversion opportunities but those that do tend to be very good (and as Mole has figured out the mean reversion tends to be cyclical meaning it works great for a while then stops)

So what kinds of things should we be testing for? The low hanging fruit is things we all pretty much know are true and don’t have to think too much about. Mean reversion (but the above scatter plots could tell you if it exists on a one day, 2 day, or weekly timeframe), momentum (the tendency for the strongest currency or stock to continue to be strong over a given period), trend following performance following X day breakouts, trend continuation following a trend hitting moving average support, pairs trading, opening gap, earnings drift post announcements, market performance over a given time period, or on Fridays, or OPEX days, etc.

Now you are going to get a lot of dry holes in your research. It’s part of the game, for me I test 5 different ideas to find one that even has potential, and I would throw away 90% of potential systems. For example let’s test whether an upmove in the Russ-hole is predictive of an upmove tomorrow (and vice versa) or maybe predictive of a down move (mean reverting over the 1 day timeframe). This big random looking scatter plot means effectively there is no relationship between the data, no exploitable effects or correlations, in simple terms it’s all bullshit.

dry hole

So, for shits and giggles lets test two different theories, which I’m  hoping to work into two potential systems over the next few days.

System 1 hypothesis. Trend Following Systems buy breakouts, typically 50 day breakouts, and currently approximately $300 billion dollars is traded on them. Since trend following systems have win rates of between 25%-35% it stands to reason there will be a pullback effect after breaking out. We have all seen breakouts that continue and go forever, and breakouts that fail. The hypothesis we want to test is

Is there money to be made shorting 50 day breakouts for a mean reversion scalp?

(this is not dissimilar to Linda Raschke’s Turtle Soup setup which Mole used to trade, back in the day)

System 2 Hypothesis: The strongest currency pair on any given day should continue to trend over the following days. 

Now, I have literally no idea whether these concepts are an edge worth pursuing, but I’m eager to find out. And if one of them is an edge, together we will build a system out of it, and hopefully you guys can work it into something tradeable in the comment section and we can backtest it into a crowdsourced evilspeculator system, ready for prime time.

As for the markets today. I have one tasty setup for you, and some commentary on the indexes. It is important to notice that a massive bar at the end of a long running low volatility trend indicates capitulation, and time to start looking at both short and longside again. Here we have 4 days of pullback, with each successive day being more overlapping and smaller range than the previous day. The obvious conclusion is that the bears are running out of steam, and the bulls are preparing for another assault on fresh highs. I have no idea whether this assault will be successful, and personally I think the easy phase of this move is done.

es aud

In retrospect I’m quite chuffed at this exit, even if it was only for a puny 3 contracts. Still, better to be lucky than good 😉

lucky

Questions about the scatter plots in the comment section.

Scott


About The Author
Scott
  • Yoda

    I like regression. I also liked the transition from proper English to Scott’s English.
    Stockcharts has a cool Slope (linear regression) tool.
    http://stockcharts.com/h-sc/ui?s=AAPL&p=D&b=5&g=0&id=p45867294768
    I like the setting on the very short tick frame. Kind of helping to see if a pullback should be bought.

  • ZigZag

    Good stuff. Sigh. Guess I really do have to learn to code. Was hoping to have my kids do it for me.

  • Scott Phillips

    Don’t despair – there is a hack around it. You can start with hypothesis that you already know are true, and there are quite a few of those.

  • Scott Phillips

    It’s not a question of like or don’t like :-) It’s a question of objectively “is it true or not?” And if it’s true, how true is it?

    We spend a lot of time saying things like “I like this indicator, I like envelopes, keltners, bollingers, whatever”. We don’t spend nearly enough time asking the hard question.

    Objectively, am I wasting my fucking time here or is it real, given that my brain constantly lies to me?

  • Zsolt Soczó

    Very good article, thanks for it.
    I have some systems running live based on the inside day idea you wrote, but I also feel uneasy because the edge is thin and the sample size small, so I never know if I curve fitted or not.
    My question is how would you make a scatter plot for the ID alg? It enters on stop so it needs some logic to simulate, in excel u cannot do this easily. I suppose you would backtest it in some backtester wo filters, stops, pts using oed exits, export the trades, merge it with 0 trades for not ID days to make a prediction for each day and make a scatter plot.
    The main q is, why it is better to visualize the results in scatter plot than simply evaluating the equity curve?
    Scatter plot does not reveal if the edge is fading in time because it mix all data for the regression, but an equity curve shows if the edge is dimishing.

  • Scott Phillips

    Ideally you should test it without stop, just take the entry and measure the positive or negative change in percent N bars after entry. Since the average trade time for an IP is 1.8 bars there is a logical place to start. Test the gain or loss on the bar of entry against the subsequent bar.

    Test ONE thing at a time, like a scientist. True or false that hypothesis and then move on.

    The problem with viewing your equity curve as relevant is you are testing both the fundamental concept (an inside period is a breakout from a lower timeframe trading range at its heart), also the stop, target, and breakeven logic. You are testing many different things together at the same time.

    The particular IP exit logic most people use (protect R) is quite clever, and makes up for the thinness of the edge. You simply do not know how much of performance is due to a clever exit algo, and how much is due to the inherent edge.

    Tested against a random brownian motion the IP will have significant periods of outperformance.

    This is the inherent and unfixable problem with Ivan’s whole approach basically. We used to produce impressive backtests and lots of walk forward tests which never matched up in the real world. The systems would perform great for a while and completely stop. I’ve spoken to a lot of his students they ALL say the same things. My system backtest looked great, it started off well, why am I in a 30R drawdown when I never saw it on the backtest?

    I would make a scatter plot with the X axis being gain or loss on the bar it breaks out of an inside period (in pips if you are testing a single market, percent if you testing multiple) and the Y axis being gain or loss on the subsequent bar. It should look like one of the scatter plots above and you then have something to work with.

  • Scott Phillips

    Actually I went ahead and did it for you, I’ll cover it in a post the day after tomorrow, and share the data in a public spreadsheet :-)

    Honestly it didn’t take long

  • Zsolt Soczó

    Thank you, I’m eagerly waiting for it. :)

    Programming and testing is not a problem for me at all (I wrote my entire system from ground up), finding real and persistent edges are. :)

  • Scott Phillips

    I only tested 60m bars from March to now, so only 13 data points. But it’s enough for you to get the idea and automate the rest if you are a programmer. It should actually be trivial for you :-)

  • http://evilspeculator.com Sir Mole III

    I fixed the order so it’s first in line now.

  • http://evilspeculator.com Sir Mole III

    The problem is that human beings often see things because they want to.

  • http://www.captainboom.com/ captainboom

    Scott, is it safe to assume you will be using the data available from Quantopia, and learning Python for the coding? Any other data sources you recommend? Free is always nice, though I’m aware you sometimes get what you pay for…

  • Ronebadger

    Thanks Scott, I’ll have to cut out some time to go thru all this…right now, my brain hurts from skimming it. I know a lot of time went into this.

  • ridingwaves

    I need to digest too, my brains are fried from work…
    Thanks Scott, the scatter plot brings back memories….I will have to do some re-learning here

  • StockTalker

    Signal getting weaker every day.

  • ridingwaves

    this could be the 10 yr auction drain, the 2350 spy gap seems ripe but only the mm’s know

  • StockTalker

    God, say it ain’t so:)

  • Mark Shinnick

    Vol has been firming. went longer this morning. Stops tightening.

  • BobbyLow

    Great stuff as usual Scott.

    Thanks for the Props. :) And special thanks for helping me dump a lot of my old baggage that led to improvements in discipline and dealing with day to day emotions that are part of this business.

  • Mark Shinnick

    Standing aside vol for time being.

  • StockTalker

    Rarely do I see price above VWAP with the signal below the signal line.

  • Tomcat

    Huge build in inventory for Crude and futures up…correlation you ask???

  • StockTalker

    She’s gana slip

  • BobbyLow

    After a shitty start to 2017, what a difference the last 8 trading days have made. Miners via DUST, Nutty Gas via UGAZ and my newest horse Silver via DSLV appear to have really turned things around.

    Hi Ho Silver Away! :)

    https://www.youtube.com/watch?v=rZIqg0gBcYM

  • StockTalker

    Down trend day by indicators on the signal.

  • ridingwaves

    don’t jinx it :), your now at a very important juncture, gap filled, GG had a chart that might be of interest in post a couple days ago…..congrats on trade
    https://s9.postimg.org/5d29c6asf/dslv_quick_look.png

  • ridingwaves

    that long sideways action on CL and chart via Jsparrow was telling, the 200 looks like a resting place…

  • BobbyLow

    Thanks RW. You’re right and I don’t want to jinx it. :)

    I’m trading off an Hourly on Silver and my Short signal came last week. If price doesn’t improve much for the rest of the day today, my Daily lens will become a short too. By rule this is the only horse that I’ll hold through weekends. I’ll also close out prior to Fed Days with the next one happening next week on Wednesday (if my position doesn’t get stopped out through the normal process beforehand). In the mean time, I’ll wait to see what price says.

  • BobbyLow

    Maybe this move has some legs. If it breaks 50 “maybe” I’ll try to get on the MOMO bus again. But after how CL’s price has behaved for the past 10 Weeks, a $2 move ain’t enough. My gut says that Crude should be trading at $35 or below. It also says that OPEC won’t be able to hold their coalition together and we’ll have supply coming out of their collective asses.

    However, I don’t trade my gut anymore because it’s been proven to be as reliable as OPEC. :)

  • Mark Shinnick

    Its isnt a system, but there are times when all the techs and charting are totally nulled, the option premiums are collapsed…and that’s all we have left.

  • http://gerb-reloaded.blogspot.com/ Gold_Gerb

    I won’t rain on his (BL) parade. He’s the guest of honor today!
    😉

    https://uploads.disquscdn.com/images/34c6a303ebd04121a00183821bc3f37ca23bc1ab11ee78d0387d0aa69cfd7e99.png

  • ridingwaves

    It will test 48.50….it has lost support above

  • BobbyLow

    Did you grab some SCO?

  • ridingwaves

    whose parade….?

  • ridingwaves

    my stop was hit yesterday on UCO, sidelines right now as I have some bio symbols heating up..

  • http://gerb-reloaded.blogspot.com/ Gold_Gerb

    Bobby’s.

    Horses dressed up in silver are very popular in Scottsdale, this time of year.

    https://www.experiencescottsdale.com/event/64th-annual-parada-del-sol-rodeo/360933/

  • Mark Shinnick

    I think this is the day gold bugs were looking for reentry.

  • ridingwaves
  • StockTalker

    Better off shorting S&P at this point.

  • StockTalker

    Distribution day, looks like the worm might finally turn.

  • BobbyLow

    In the past I would impulsively jump all over this. But FBOW, I’m going to see if there is any follow through tomorrow or any retracement. Either way might be OK because if there’s follow through then perhaps this move has legs. If there is a retracement and I still decide to get in, I’ll get a better price.

  • BobbyLow

    Maybe so but for whatever reason the S&P has been an achilles heel for me. I’m better off messing with a few heads of cattle from the various sectors of the market instead of the entire herd. In other words, I suck at trading major Indices such as the SPX, NDX and RUT. :)

  • Mark Shinnick

    At the moment…supportive to vol.

  • StockTalker

    We want a lot of volume on down days.

  • Mark Shinnick

    Its interesting that once the gap got filled…no buyers really showed up.

  • StockTalker

    That is a good thing, bears will pounce on this.

  • StockTalker

    Nice looking candles

  • http://gerb-reloaded.blogspot.com/ Gold_Gerb

    Significant daily line crossed on energy..
    {ignore the news}

    [XLE]
    https://uploads.disquscdn.com/images/e6c9ad25a95697167751f40cfc3e2076156d5a569a36b34a922f1ecade221eb2.png
    -GG

  • BobbyLow

    Looks like Crude might break 50 today. My human nature is telling me that I gave up on this horse one trade too soon. (Funny we were just talking about this yesterday.) Fortunately I swapped out Crude for 3 other horses that are all running fine. So it is what it is. At times like this, I try to remember what a big hitter told me many, many years ago and that is “never worry about the deal you didn’t make”.

  • ridingwaves

    yesterday the Saudi oil minister said the worst is behind for oil sector after oil producer country meeting in Houston…..maybe it will reverse when he says were screwed…:)

  • StockTalker

    Ouch

  • ridingwaves

    here is that gap I’ve mentioned a couple times…
    https://s17.postimg.org/o32ihga4v/I_spy_a_gap.png

  • Scott Phillips

    It is what it is :) I was stopped out of CL short for breakeven 2 days ago, banked a 1R scalp on my mean reversion system the day before, and yesterday went short again on break of the daily low. Banked half profit at 50.50
    https://snag.gy/Ka9tAl.jpg

  • Scott Phillips

    Don’t forget that SPY isn’t actually a real thing. It’s a derivative, and most of the time the gaps in it are a function of the trading hours.

    Doing technical analysis on ETF’s is like smearing vaseline on a pair of sunglasses, you get the main idea pretty well, but the fine detail is untrustworthy :-)

  • Scott Phillips

    No, thats literally the dumbest thing I’ve heard all day, and I’ve been on facebook already today.

    We don’t short the strongest markets, nor the strongest stocks. We short the shitty markets, and the shitty stocks.

  • ridingwaves

    I like that smearing quote….
    Spx shows the gap but it’s much tighter…and so close it might not be

  • Scott Phillips

    It’s been a real pleasure watching you turn into the solid pro you are :-) The credit all belongs to you, and I wish others would follow your example and build systems, test those systems for expectancy, then trade those systems keeping proper records.

    People don’t get that that’s ALL THERE IS TO IT!

  • Scott Phillips

    Yeah I had to do some relearning too.

  • CandleStickEmUpper

    good quote. ETF’s suck

  • Scott Phillips

    Good data is expensive :) CSI data has a good data package for a reasonable price. Also I’m interested in testing quandl

    Right now I use IQfeed and Kinetick data off ninjatrader. I’m really just in the beginner phase with quantopian, but I like what I see so far.

  • Scott Phillips

    Tanks mang

  • Scott Phillips

    Or it might be a different kind of gap, a micro gap (open above the previous close but below the previous high) which is a totally different thing which I’m going to cover today.

    Also, the idea that GAPS MUST BE FILLED, is nonsense from the 1970’s. There is a strong gap fill effect same day, and that effect weakens as time goes on.

    The idea that gaps somehow act as magnets to price months later is nothing short of witchcraft

  • Yoda

    Mole, if you read this, please extend your vacations.

  • ridingwaves

    I agree with your statement but lately a lot of them have filled, so my bias is leaning that way…i’m using the 25 day green line as part of the bias..

  • Yoda

    no they look for an entry

  • Scott Phillips

    Yeah thats the thing. We see what we want to see. Is it a “gap filling” or just price moving all over the chart and having fuck all to do with the gap?

    Logically which do you think is more likely?

    Price does go up and go down pretty much every day, yaknow

  • Ronebadger

    “the best gaps don’t fill…”

  • ridingwaves

    point taken….the market is a pogo stick with a heavy rider…

  • Scott Phillips

    More to the point we ALL see what we are looking for, we bred for it. If you put any random indicator on your chart, inside of 2 weeks you will swear its an edge. Gaps are no different :)

  • Scott Phillips

    NEW POST

  • http://evilspeculator.com Sir Mole III

    That’s a bit insulting.

  • Yoda

    Didn’t mean to offend you. it is just that I would love my short SPX trade to run for a little longer.

  • Mark Shinnick

    Its so true, such as in a range environment we can arbitrarilydefine with 1 or 2 deviations well within an average daily true range.

  • Ladywandering

    Just read your post. Can you post a table for one of your plots above. thanks

  • Kidd Cudi

    good post