Author:
Automating Technical Analysis & Systematic BackTesting of Trading Strategies with Python
In this article I will be using some awesome packages to show you how simple it is to get started with automating the generation of technical analysis features. I will then go on to use these features to generate some simple systematic trading strategies that I will then backtest. This is about the simplest workflow that I have seen for this. I hope you will be pleasantly surprised.
Techincal Analysis
Technical Analysis is the study of investments using prices, volumes and other derived data point histories. As a time-honoured discipline, over the years many indicators have been produced from all parts of the globe. Long the preserve of human eyeballs - the advent of computing in finance has changed the technical analysis landscape with a new level of rigour - leading to the emergence of Systematic Trading and Investment - now well established and the emergent field of AI-first finance - which could leverage such features for Machine Learning. The spread of Python has greatly democratised such pursuits - making them easier than ever to master.
Sections
Implement Candlestick Pattern Recognition
Pre-requisites:
Refinitiv Eikon / LSEG Workspace with access to Eikon Data APIs (Free Trial Available)
Python 2.x/3.x
Required Python Packages: Refinitiv Data Libraries, pandas, numpy, matplotlib, TA-Lib, pandas_ta, mplfinance, backtrader
Import all the packages we need
import refinitiv.data.eikon as ek
import talib
import pandas_ta as pta
import backtrader as bt
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib as mpl
import mplfinance as mpf
import datetime as dt
import numpy as np
import warnings
warnings.simplefilter("ignore")
import configparser
cfg = configparser.ConfigParser()
cfg.read('rdp.cfg',encoding='utf-8')
%matplotlib inline
plt.style.use("dark_background")
mpl.style.use("dark_background")
ek.set_app_key(cfg['eikon']['app_key'])
#ek.set_app_key('YOUR APP KEY HERE')
Get our data - one Line API call
Here we use our get_timeseries function to return us hourly OHLCV for cable (GBP=). As FX doesn't carry volume directly - we can use count as a surrogate. We then rename the columns so they can work with charting and backtesting software downstream. Note the dataframe comes back already indexed on timestamp.
df1= ek.get_timeseries('GBP=',fields=['OPEN','HIGH','LOW','CLOSE','COUNT'],start_date='01-10-2023',end_date='06-27-2023',interval='hour') #,corax='adjusted'
df1.columns=['open','high','low','close','volume']
df1
Date | open | high | low | close |
volume |
---|---|---|---|---|---|
2023-01-10 00:00:00 | 1.2181 | 1.2189 | 1.2175 | 1.2182 | 3474 |
2023-01-10 01:00:00 | 1.2183 | 1.2197 | 1.2162 | 1.2175 | 5575 |
2023-01-10 02:00:00 | 1.2172 | 1.2184 | 1.2163 | 1.2179 | 7952 |
2023-01-10 03:00:00 | 1.2178 | 1.2186 | 1.2154 | 1.2167 | 8144 |
2023-01-10 04:00:00 | 1.2167 | 1.2175 | 1.2156 | 1.2166 | 7134 |
... | ... | ... | ... | ... | ... |
2023-06-26 20:00:00 | 1.2718 | 1.2721 | 1.2708 | 1.271 | 2247 |
2023-06-26 21:00:00 | 1.2709 | 1.2714 | 1.2707 | 1.2711 | 1582 |
2023-06-26 22:00:00 | 1.2712 | 1.2716 | 1.2703 | 1.2709 | 1027 |
2023-06-26 23:00:00 | 1.271 | 1.2714 | 1.2708 | 1.2713 | 2362 |
2023-06-27 00:00:00 | 1.2712 | 1.2713 | 1.2707 | 1.271 | 1683 |
2930 rows x 5 columns |
df1.dropna(how="any", inplace=True)
len(df1)
2930
Pandas TA package is very useful tool to create various technical analysis features with ease
Types of Indicator
Cycles(1), Momentum(41), Overlap(33), Performance(3), Statistics(11), Trend(18), Utility(5), Volatility(14), Volume(15) as well as Candlestick Patterns (64) are provided by the package ready for you to implement - and as we will see implementation is a breeze. The breadth of indicators means that you can experiment to your hearts content with combinations of indicators etc. We can look at what these are as below:
df1.ta.indicators()
Pandas TA - Technical Analysis Indicators - v0.3.14b0
Total Indicators & Utilities: 205
Abbreviations:
aberration, above, above_value, accbands, ad, adosc, adx, alma, amat, ao, aobv, apo, aroon, atr, bbands, below, below_value, bias, bop, brar, cci, cdl_pattern, cdl_z, cfo, cg, chop, cksp, cmf, cmo, coppock, cross, cross_value, cti, decay, decreasing, dema, dm, donchian, dpo, ebsw, efi, ema, entropy, eom, er, eri, fisher, fwma, ha, hilo, hl2, hlc3, hma, hwc, hwma, ichimoku, increasing, inertia, jma, kama, kc, kdj, kst, kurtosis, kvo, linreg, log_return, long_run, macd, mad, massi, mcgd, median, mfi, midpoint, midprice, mom, natr, nvi, obv, ohlc4, pdist, percent_return, pgo, ppo, psar, psl, pvi, pvo, pvol, pvr, pvt, pwma, qqe, qstick, quantile, rma, roc, rsi, rsx, rvgi, rvi, short_run, sinwma, skew, slope, sma, smi, squeeze, squeeze_pro, ssf, stc, stdev, stoch, stochrsi, supertrend, swma, t3, td_seq, tema, thermo, tos_stdevall, trima, trix, true_range, tsi, tsignals, ttm_trend, ui, uo, variance, vhf, vidya, vortex, vp, vwap, vwma, wcp, willr, wma, xsignals, zlma, zscore
Candle Patterns:
2crows, 3blackcrows, 3inside, 3linestrike, 3outside, 3starsinsouth, 3whitesoldiers, abandonedbaby, advanceblock, belthold, breakaway, closingmarubozu, concealbabyswall, counterattack, darkcloudcover, doji, dojistar, dragonflydoji, engulfing, eveningdojistar, eveningstar, gapsidesidewhite, gravestonedoji, hammer, hangingman, harami, haramicross, highwave, hikkake, hikkakemod, homingpigeon, identical3crows, inneck, inside, invertedhammer, kicking, kickingbylength, ladderbottom, longleggeddoji, longline, marubozu, matchinglow, mathold, morningdojistar, morningstar, onneck, piercing, rickshawman, risefall3methods, separatinglines, shootingstar, shortline, spinningtop, stalledpattern, sticksandwich, takuri, tasukigap, thrusting, tristar, unique3river, upsidegap2crows, xsidegap3methods
Lets see how easy it is to create some Bollinger Bands
Here we will use a popular indicator in the volatility catergory - Bollinger Bands. First we will make a copy of our base price dataframe and then append the bollinger bands feature to the new dataframe. We pass some parameters such as window length of 50 periods, using an simple moving average. We can see the BBL (lower), BBM (mid) and BBU (upper) features, amongst others added to the right of the frame.
df2 = df1.copy()
df2.ta.bbands(length=50, std=2, mamode="sma", ddof=0, append=True)
df2
Date | open | high | low | close | volume | BBL_50_2.0 | BBM_50_2.0 | BBU_50_2.0 | BBB_50_2.0 | BBP_50_2.0 |
---|---|---|---|---|---|---|---|---|---|---|
2023-01-10 00:00:00 | 1.2181 | 1.2189 | 1.2175 | 1.2182 | 3474 | NaN | NaN | NaN | NaN | NaN |
2023-01-10 01:00:00 | 1.2183 | 1.2197 | 1.2162 | 1.2175 | 5575 | NaN | NaN | NaN | NaN | NaN |
2023-01-10 02:00:00 | 1.2172 | 1.2184 | 1.2163 | 1.2179 | 7952 | NaN | NaN | NaN | NaN | NaN |
2023-01-10 03:00:00 | 1.2178 | 1.2186 | 1.2154 | 1.2167 | 8144 | NaN | NaN | NaN | NaN | NaN |
2023-01-10 04:00:00 | 1.2167 | 1.2175 | 1.2156 | 1.2166 | 7134 | NaN | NaN | NaN | NaN | NaN |
... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
2023-06-26 20:00:00 | 1.2718 | 1.2721 | 1.2708 | 1.271 | 2247 | 1.269292 | 1.271990 | 1.274688 | 0.424205 | 0.316526 |
2023-06-26 21:00:00 | 1.2709 | 1.2714 | 1.2707 | 1.2711 | 1582 | 1.269311 | 1.271922 | 1.274533 | 0.410630 | 0.342616 |
2023-06-26 22:00:00 | 1.2712 | 1.2716 | 1.2703 | 1.2709 | 1027 | 1.269337 | 1.271848 | 1.274359 | 0.394936 | 0.311267 |
2023-06-26 23:00:00 | 1.271 | 1.2714 | 1.2708 | 1.2713 | 2362 | 1.269360 | 1.271792 | 1.274224 | 0.382411 | 0.398838 |
2023-06-27 00:00:00 | 1.2712 | 1.2713 | 1.2707 | 1.271 | 1683 | 1.269407 | 1.271722 | 1.274037 | 0.364007 | 0.344032 |
2930 rows x 10 columns |
We can now easily plot these as one would in a Workspace chart - its not as polished as the Workspace chart - but this is for programmatic usage mainly as opposed to in-depth interactive detailed visualisation.
df2[['close','BBL_50_2.0','BBU_50_2.0']].plot(figsize=(18,15))
Candlestick pattern recognition
As we noted earlier there are 64 candlestick patterns that can be identified in both bullish and bearish configurations. Bullish structures are indicated with +100 and bearish structures are indicated with -100. In our case we are using hourly data so we are notified in the hourly row when the signal was generated. Again we simply make a copy of our original prices frame and then append all candlestick patters to that dataframe. One can of course select only the indicators you are interested in.
df3 = df1.copy()
df3.ta.cdl_pattern(name="all",append=True)
df3
Date | open | high | low | close | volume |
CDL_2CROWS | CDL_3BLACKCROWS | CDL_3INSIDE | CDL_3LINESTRIKE | CDL_3OUTSIDE | ... | CDL_SPINNINGTOP | CDL_STALLEDPATTERN | CDL_STICKSANDWICH | CDL_TAKURI | CDL_TASUKIGAP | CDL_THRUSTING | CDL_TRISTAR | CDL_UNIQUE3RIVER | CDL_UPSIDEGAP2CROWS | CDL_XSIDEGAP3METHODS |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
2023-01-10 00:00:00 | 1.2181 | 1.2189 | 1.2175 | 1.2182 | 3474 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
2023-01-10 01:00:00 | 1.2183 | 1.2197 | 1.2162 | 1.2175 | 5575 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
2023-01-10 02:00:00 | 1.2172 | 1.2184 | 1.2163 | 1.2179 | 7952 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
2023-01-10 03:00:00 | 1.2178 | 1.2186 | 1.2154 | 1.2167 | 8144 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
2023-01-10 04:00:00 | 1.2167 | 1.2175 | 1.2156 | 1.2166 | 7134 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
2023-06-26 20:00:00 | 1.2718 | 1.2721 | 1.2708 | 1.271 | 2247 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
2023-06-26 21:00:00 | 1.2709 | 1.2714 | 1.2707 | 1.2711 | 1582 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
2023-06-26 22:00:00 | 1.2712 | 1.2716 | 1.2703 | 1.2709 | 1027 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | -100.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
2023-06-26 23:00:00 | 1.271 | 1.2714 | 1.2708 | 1.2713 | 2362 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
2023-06-27 00:00:00 | 1.2712 | 1.2713 | 1.2707 | 1.271 | 1683 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
2930 rows × 67 columns
It can be a bit difficult to see when these signals are being generated from large dataframes - so you can check the min and max readings to see if any bullish or bearish signals are present.
df3.describe()
Feature | open | high | low | close | volume | CDL_2CROWS | CDL_3BLACKCROWS | CDL_3INSIDE | CDL_3LINESTRIKE | CDL_3OUTSIDE | ... | CDL_SPINNINGTOP | CDL_STALLEDPATTERN | CDL_STICKSANDWICH | CDL_TAKURI | CDL_TASUKIGAP | CDL_THRUSTING | CDL_TRISTAR | CDL_UNIQUE3RIVER | CDL_UPSIDEGAP2CROWS | CDL_XSIDEGAP3METHODS |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
count | 2930.000000 | 2930.000000 | 2930.000000 | 2930.000000 | 2930.000000 | 2930.0 | 2930.0 | 2930.000000 | 2930.000000 | 2930.000000 | ... | 2930.000000 | 2930.000000 | 2930.0 | 2930.000000 | 2930.0 | 2930.000000 | 2930.000000 | 2930.0 | 2930.0 | 2930.000000 |
mean | 1.233737 | 1.234784 | 1.232719 | 1.233752 | 4231.047440 | 0.0 | 0.0 | 0.068259 | -0.034130 | -0.477816 | ... | 4.846416 | -0.273038 | 0.0 | 1.979522 | 0.0 | -0.034130 | 0.102389 | 0.0 | 0.0 | 0.000000 |
std | 0.021983 | 0.021885 | 0.022104 | 0.021999 | 1986.121237 | 0.0 | 0.0 | 7.838990 | 4.888535 | 16.928051 | ... | 49.064937 | 5.219052 | 0.0 | 13.931976 | 0.0 | 1.847422 | 4.887582 | 0.0 | 0.0 | 3.695475 |
min | 1.181000 | 1.181900 | 1.180200 | 1.181000 | 1.000000 | 0.0 | 0.0 | -100.000000 | -100.000000 | -100.000000 | ... | -100.000000 | -100.000000 | 0.0 | 0.000000 | 0.0 | -100.000000 | -100.000000 | 0.0 | 0.0 | -100.000000 |
25% | 1.215525 | 1.216900 | 1.214400 | 1.215525 | 2877.000000 | 0.0 | 0.0 | 0.000000 | 0.000000 | 0.000000 | ... | 0.000000 | 0.000000 | 0.0 | 0.000000 | 0.0 | 0.000000 | 0.000000 | 0.0 | 0.0 | 0.000000 |
50% | 1.238000 | 1.239050 | 1.236950 | 1.237900 | 4140.500000 | 0.0 | 0.0 | 0.000000 | 0.000000 | 0.000000 | ... | 0.000000 | 0.000000 | 0.0 | 0.000000 | 0.0 | 0.000000 | 0.000000 | 0.0 | 0.0 | 0.000000 |
75% | 1.247475 | 1.248500 | 1.246300 | 1.247375 | 5473.000000 | 0.0 | 0.0 | 0.000000 | 0.000000 | 0.000000 | ... | 0.000000 | 0.000000 | 0.0 | 0.000000 | 0.0 | 0.000000 | 0.000000 | 0.0 | 0.0 | 0.000000 |
max | 1.283200 | 1.284800 | 1.282500 | 1.283300 | 10525.000000 | 0.0 | 0.0 | 100.000000 | 100.000000 | 100.000000 | ... | 100.000000 | 0.000000 | 0.0 | 100.000000 | 0.0 | 0.000000 | 100.000000 | 0.0 | 0.0 | 100.000000 |
Lets use Matplotlib Finance package to create some candlestick charts with ease
We can use the candlestick charts to help us visualise when these candlestick patterns fire - first lets create a candlestick chart and then overlay a candlestick signal - in our case the CDL_3INSIDE (3 inside candles pattern - which is a compression signal which can present in both bullish (+100) and bearish (-100) variants.
df3 = df3.astype(float)
mpf.plot(df3, type='candle')
apdict = mpf.make_addplot(df3['CDL_3INSIDE'])
mpf.plot(df3,volume=True,addplot=apdict)
Implement backtesting of a strategy using BackTrader package
So we have seen how we can automate the creation of Techincal Analysis indicators very easily from a dataframe of prices. The ease with which we executed this is kind of remarkable. The next stage we can go to - is to use these indicators to create a trading strategy and to then test its efficacy through backtesting. This is also relatively easy to accomplish using the BackTrader package. First lets make a copy of our original dataframe.
df4 = df1.copy()
Set up a simple SMA crossover strategy using built-in strategy
The BackTrader package uses a base strategy class which you can use to inherit from to build your own custom strategy. In our case we will be implementing 2 trading strategies firstly a simple moving average crossover (MA XOver) strategy and then a slightly more complex combinatorial strategy where we only take signals in the direction of the trend.
smaCross strategy
First you can see this is composed of a list of parameter (in our case the window length for the moving averages). Next an init section which defines both the indicators from the bt.ind collection and also the crossover event itself. Note that the cross over event is bidirectional - though one can change this etc. Finally we have the trading logic section which uses the previously defined event signal (crossover signal) and combines it with our position information to inform an action. In our case we have defined to actions -
* If we do not have a position and we have a bullish cross (ie fast sma crosses slow sma from below) then take a position
* If we do have a position and we have a bearish cross (ie fast sma crosses slow sma from above) then close the position
Note - we don't allow shorting in this first strategy for simplicities sake. We will include shorting in the second of our strategies.
class smaCross(bt.Strategy):
# list of parameters which are configurable for the strategy
params = dict(
pfast=50, # period for the fast moving average
pslow=200 # period for the slow moving average
)
def __init__(self):
sma1 = bt.ind.SMA(period=self.p.pfast) # fast moving average
sma2 = bt.ind.SMA(period=self.p.pslow) # slow moving average
self.crossover = bt.ind.CrossOver(sma1, sma2) # crossover signal
def next(self):
if not self.position and self.crossover > 0: # not in the market
self.buy(size=100)
elif self.position and self.crossover < 0: # in the market & cross to the downside
self.close() # close long position
Candle Mix Stategy
Here we extend our basic smaCross strategy by say introducing a candlestick pattern - in this case the 3Inside candlestick pattern.
class cdlmix(bt.Strategy):
# list of parameters which are configurable for the strategy
params = dict(
pfast=50, # period for the fast moving average
pslow=100 # period for the slow moving average
)
def __init__(self):
self.cdl3 = bt.talib.CDL3INSIDE(self.data.open,self.data.high,self.data.low, self.data.close)
self.sma1 = bt.ind.SMA(period=self.p.pfast) # fast moving average
self.sma2 = bt.ind.SMA(period=self.p.pslow) # slow moving average
self.crossover = bt.ind.CrossOver(self.sma1, self.sma2) # crossover signal
def next(self):
if not self.position:
if self.sma1 > self.sma2 and self.cdl3 == 100: # not in the market
self.buy(size=100)
if self.sma1<self.sma2 and self.cdl3 == -100:
self.sell(size=100)
elif self.position.size > 0 and (self.crossover <0 or self.cdl3 == -100): # in the market & cross to the downside
self.close() # close long position
#self.sell(size=100) # Open short
elif self.position.size < 0 and (self.crossover >0 or self.cdl3 == 100): # in the market & cross to the downside
self.close() # close short position
self.buy(size=100) # Open long
Now that we have our two strategies defined - we can pass it to our broker object which we configure to run the strategy we just created. From below we can see that the first section is where we initialise the bt.Cerebro object - setting our initial cash level and also commission rates. Next we add the strategy we just created and then define a data object which we wire up to our pandas dataframe, and specify the from and to dates as datetime objects and as we are using hourly bars we set the timeframe to be minutes with a compression rate of 60. We then add the data to the broker object and run the backtest and finally we format the standard chart output generated.
For easy multiple strategy tests - I have created a generalised brokerObject function from the basic package documentation which takes a number of parameters in order to create a valid backtest of a valid strategy and just configured it to return the cerebro.plot object which contains all the results.
# create a function for this which takes a strategy_name, df, date_from, date_to & time_frame & compression as an input
def brokerObject(strategy_name,data_frame,from_date,to_date,time_frame,compression):
# initialize backtrader broker
cerebro = bt.Cerebro(stdstats=True)
cerebro.broker.setcash(1000)
cerebro.broker.setcommission(commission=0.001)
# add strategy
cerebro.addstrategy(strategy_name)
# wire up bt.feeds.PandasData to data_frame, set timeperiod, timeframe and also compression
data = bt.feeds.PandasData(dataname=data_frame,fromdate=from_date,todate=to_date,
timeframe=time_frame,compression=compression)
cerebro.adddata(data)
# run backtest
res = cerebro.run()
strat = res[0]
#prepare plots
mpl.rcParams['font.sans-serif']=['DejaVu Sans']
mpl.rcParams['axes.unicode_minus']=False
mpl.rcParams['figure.figsize']=[18, 16]
mpl.rcParams['figure.dpi']=200
mpl.rcParams['figure.facecolor']='w'
mpl.rcParams['figure.edgecolor']='k'
return cerebro.plot(style='candle',iplot=False,width=30,height=30,start=from_date, end=to_date)
brokerObject(smaCross,df4,dt.datetime(2023, 1, 10),dt.datetime(2023, 6, 26),bt.TimeFrame.Minutes,60)
brokerObject(cdlmix,df4,dt.datetime(2023, 1, 10),dt.datetime(2023, 6, 26),bt.TimeFrame.Minutes,60)
In this article we have seen how we can very easily generate Technical Analysis features using Pandas_TA / TA_Lib packages. These work seamlessly with the output from our Eikon Data API and are simply appended to the right of the dataframe. We have also seen how we can eyeball these indicators using basic charting in Python. In particular we have shown how we can plot candlestick patterns firing on a chart with real simplicity.
We then went on to take things a little further by creating some simple trading strategies using our indicators in the strategy template of the backtrader package and then wired that up to the backtrader broker object that conducted the backtest for us and outputted a series of plots that visualised the strategy and sig-gen, P&L and such.
This is really the simplest workflow I have seen for backtesting trading strategies - and whilst there are undoubtably more sophisticated packages and approaches - particularly for live streaming strategies - I think this is a great introductory approach and worthy of your attention.
Further Resources
For Content Navigation in Eikon / Workspace - please use the Data Item Browser Application: Type 'DIB' into Search Bar.
For Python/ AI/ML / Quant Finance / Algo Trading training
Books:
For Technical Analysis training & consultancy - Trevor Neil
Meetup Group