BACK-TEST METHODOLOGYWe use JSE historical price and fundamentals data provided by ProfileData (www.profile.co.za) for all our backtests. We use their excellent ShareMagicPRO software, coupled with screen rules we have developed in-house to automatically screen for candidate shares for various strategies. All back-tests use "point in time" data to reflect exactly what an investor would have witnessed in the form of publicly available data, to avoid look-ahead bias. The data used includes suspended issues, even of companies such as SAAMBOU that were liquidated. However survivorship bias is not completely eliminated in the tests since de-listings and the effects thereof were not included in the tests. Most de-listings are a result of acquisitions, swaps etc. and do not involve shareholders losing funds. However de-listings as a result of liquidations and/or bankruptcies were not included in tests of strategy performance.
We are working with ProfileData to solve the complex issue of how to incorporate previous de-listings into the data tests to provide more accurate and conservative results with less survivorship bias. However, most of the strategies we work with have such a significant out-performance of the all-share that we don't believe it will have a material effect. Additionally, most of the backtests we are doing are merely to confirm trends already extensively back-tested on overseas markets with data devoid of survivorship bias. So the general trend confirmation is more important than exact return calculations.
Additionally, we benchmark strategy performance against an equal weighted JSE index called GROUP as well as the ALSI. GROUP is subject to the same survivorship bias as the strategy being back-tested and so the relative performance of the strategy against GROUP is more the point of focus as opposed to the exact return.
We also track all strategies in real time through the Strategy Blogs to confirm historical data, and the blogs are certainly devoid of survivorship bias, forming the basis for Forward Testing.
Backtests are done on 20 years of JSE data and encompass many rolling sets of 1, 3 and 5 year data to confirm a strategy trend. Results from the different sets are averaged together to derive overall strategy performance and minimise anomalies and the effects of incorrect database entries. International research says that 10 sets of data are statistically significant for deriving trend conclusions.
Breadth data forms an important part of our service offering. All breadth data used in historical comparisons, calibrations of thresholds and back-testing of SwissClock and BPI timing systems goes back 13 years and includes advance/decline and other breadth data of all JSE main board companies listed at that time. Survivorship bias is thus not an issue for our breadth data. As per international norms, we use ratio-adjusted breadth data to cater for the effects of changing number of shares that are listed over time.