DynamicHedge recently introduced a new service called “alpha curves”: the main idea is to find patterns in returns after certain events, and present the most frequently occurring patterns. In their own words, alpha curves “represent a special blend of uniqueness and repeatability”. Here’s what they look like, ranked in order of “pattern dominance”. According to them, they “use different factors other than just returns”. We can speculate about what other factors go into it, possibly something like maximum extension or the timing of maxima and minima, but I’ll keep it simple and only use returns.
In this post I’ll do a short presentation of dynamic time warping, a method of measuring the similarity between time series. In part 2 we will look at a clustering method called K-medoids. Finally in part 3 we will put the two together and generate charts similar to the alpha curves. The terminology might be a bit intimidating, but the ideas are fundamentally highly intuitive. As long as you can grasp the concepts, the implementation details are easy to figure out.
To be honest I’m not so sure about the practical value of this concept, and I have no clue how to quantify its performance. Still, it’s an interesting idea and the concepts that go into it are useful in other areas as well, so this is not an entirely pointless endeavor. My backtesting platform still can’t handle intraday data properly, so I’ll be using daily bars instead, but the ideas are the same no matter the frequency.
So, let’s begin with why we need DTW at all in the first place. What can it do that other measures of similarity, such as Euclidean distance and correlation can not? Starting with correlation: one must keep in mind that it is a measure of similarity based on the difference between means. Significantly different means can lead to high correlation, yet strikingly different price series. For example, the returns of these two series have a correlation of 0.81, despite being quite dissimilar.
A second issue, comes up in the case of slightly out of phase series, which are very similar but can have low correlations and high Euclidean distances. The returns of these two curves have a correlation of .14:
So, what is the solution to these issues? Dynamic Time Warping. The main idea behind DTW is to “warp” the time series so that the distance measurement between each point does not necessarily require both points to have the same x-axis value. Instead, the points further away can be selected, so as to minimize the total distance between the series. The algorithm (the original 1987 paper by Sakoe & Chiba can be found here) restricts the first and last points to be the beginning and end of each series. From there, the matching of points can be visualized as a path on an n by m grid, where n and m are the number of points in each time series.
Source: Elena Tsiporkova, Dynamic Time Warping Algorithm for Gene Expression Time Series
The algorithm finds the path through this grid that minimizes the total distance. The function that measures the distance between each set of points can be anything we want. To restrict the number of possible paths, we restrict the possible points that can be connected, by requiring the path to be monotonically increasing, limiting the slope, and restricting how far away from a straight line the path can stray. The difference between standard Euclidean distance and DTW can be demonstrated graphically. In this case I use two sin curves. The gray lines between the series show which points the distance measurements are done between.
DTW
Euclidean
Notice the warping at the start and end of the series, and how the points in the middle have identical y-values, thus minimizing the total distance.
What are the practical applications of DTW in trading? As we’ll see in the next parts, it can be used to cluster time series. It can also be used to average time series, with the DBA algorithm. Another potential use is k-nn pattern matching strategies, which I have experimented with a bit…some quick tests showed small but persistent improvements in performance over Euclidean distance.
If you want to test it out yourselves, there are plenty of tools out there. I’m using the NDTW .NET library. There are libraries available for R and python as well.
Sam Harris says:
Thank you for this post. Very interesting. I have been looking at Dynamic Hedge articles when he mentioned Alpha Curves for quite some time. And then I spotted your questin below the blog post and no answer from him and was just waiting for your blog post on the topic… 🙂
But please, don’t keep it simple (as you are saying: “We can speculate about what other factors go into it, possibly something like maximum extension or the timing of maxima and minima, but I’ll keep it simple and only use returns.”). Can you please explain what you mean by “…maximum extension or the timing of maxima and minima…” and possibly give an example in future posts. Thanks.
There are nice posts on tye topic here:
– http://systematicinvestor.wordpress.com/2012/01/13/time-series-matching/
– http://systematicinvestor.wordpress.com/2012/01/20/time-series-matching-with-dynamic-time-warping/
qusma says:
Hi Sam,
Those are some great links, thanks. I really like the color-coded graphs that show the matched sequences. This one is good, too: http://etfprophet.com/mrkt_analogues-model/
For practical trading purposes the two most important points of an “alpha curve” are the points of maximum drawdown and maximum run-up. Any price targets or stops would presumably be based on the magnitude of those two, and time-based exits would be based on when they occur. So I think it might make sense to use these numbers when it comes to computing the distance between two series (and thus when it comes to clustering). The more similar the extremes, the smaller the distance.
experquisite says:
Is the Dynamic Time Warping just a way of working around the artifact of calendar time? I have had good results by normalizing time according to tick count/volume as a method of reconstituting normality of returns (somewhat), but inhomogeneous delta-T increments makes many cointegration tests much more challenging, a problem I am still working on.
qusma says:
It can be used to work around the issue of calendar time, but it also has many other uses. Beyond the issue of similar series that are slightly out of phase, it can also be used to “work around” pattern similarity when the two series are at different speeds (the algorithm handles series of different lengths transparently, http://ej.iop.org/images/0957-0233/23/5/055601/Full/mst411675f1_online.jpg ), or the patterns exist on different scales.
The cointegration application seems interesting…I don’t think there’s any literature on the combination of DTW and cointegration tests but I think it could be a good match: use DTW to align the series and then run the ADF test (or whatever you use) on the aligned data…
Reverse Engineering DynamicHedge's "Alpha Curves", Part 2 of 3: K-Medoids Clustering : QUSMA says:
[…] the first part of the series we covered dynamic time warping, here we look at clustering. K-means clustering is […]
Reverse Engineering DynamicHedge’s “Alpha Curves”, Part 2 of 3: K-Medoids Clustering says:
[…] the first part of the series we covered dynamic time warping. Here we look at clustering. K-means clustering is […]
Reverse Engineering DynamicHedge’s “Alpha Curves”, Part 2.5 of 3: DTW Barycenter Averaging says:
[…] Reverse Engineering DynamicHedge’s Alpha Curves, Part 1 of 3: Dynamic Time Warping […]
Visualizing the Similarity Between Multiple Time Series : QUSMA says:
[…] It should be noted that MDS doesn’t care about how you choose to measure the distance between the time series. While I used correlations in this example, you could just as easily use a technique like dynamic time warping. […]
Reverse Engineering DynamicHedge’s Alpha Curves, Part 1 of 3: Dynamic Time Warping | Supernova Capital says:
[…] From QUSMA […]
khoeeni says:
these links are excellent. I thank the information
لامپ کم مصرف–کرکره برقی–راهبند اتوماتیک–درب پارکینگ–راهبند