I couldn’t agree more. I seem to see buys and sells for the same stocks repeatedly.
Steve,
You are right that most high return strategies are shuffling around very similar stocks (largely small and micro caps with high betas). Although our systems may not include a factor directly looking for high beta, the factors that produce big gainers tend to indirectly pick a lot of high beta stocks. High beta stocks are fine if the market is trend upward, but it is a different story when the general trend is down.
The biggest thing stopping P123 users from designing better systems is not a more effective analysis tool --- rather it is the relatively [b]SHORT DATA HISTORY[/b] currently available. Right now we have just a few months of data from a major market down turn (2001-mid 2002). To be able to develop really good systems that can work in all types of markets, we need [b]DATA from ALL TYPES OF MARKETS[/b]. At present y we have data from a moderate bull market recovery plus a tiny bit of late term bust data. Ideally we should be developing our systems using 2 or 3 bull periods (1992-94, 1996-97, 1999-2000, 2003-2006) AND 2 or 3 small and large bear periods (1990-91, 1995, 1998, 2000-2002). By far the [b]BIGGEST and BEST IMPROVEMENT[/b] for P123 users would be not more technical metrics, but a[b] LONGER DATA HISTORY[/b]. As you can tell I would like data from 1990 on, but I would be be quite happy to get it from 1995, or even just from 1997 on.
It sure would be nice to have extended data so we could test the robustness of our strategies before the next big down turn comes. Otherwise, when it does come, there will be many unhappy faces. Of course, such a future downturn would give data from a down turn so we could do tests to improve our systems -- However, I would prefer to have a longer history NOW so I can learn from the past rather than the future. Learn from the future always has a higher tuition fee than learning from the past. But we need a long data history to be able to learn from the past and thus be better prepared for the future.
Steve, I don’t mind at all when I’m buying and selling the same stocks hundreds of people are. As long as I’m buying before they’re buying and I’m selling before they’re selling, the more the merrier! 
I am certainly not against the further development of the ranking system testing tools but the first post of this thread makes me skeptical if the approach being suggested by marco is worth expending energy on. The ranking of ranking systems idea according to their linear regression fit expressed in the first post I think is dubious. Like Jerrod I think it makes assumptions that do not necessarily hold true. Danparquette has already shared his studies that implemented that and I don’t see how we’re going to come up with anything significantly better going down that road. I think it leads to the problem you are stating about the same stocks coming up again and again.
I hope the developers adopt a problem solving approach to any redesign of the ranking system testing tools. Identify shortcomings and limitations of the current setup and methodology and then fix them rather than introducing new stuff indiscriminantly. They should also be very aware of the statistical assumptions they are making that may not hold true. Optimization based on inaccurate assumptions isn’t going to get one anywhere. My guess is that the reason Danparquette’s TopFactors ranking systems aren’t as robust as we might hope is because they are overoptimized and curve fitted. They also assume a linear relationship between ranking factor and return: higher is better or lower is better. I echo Martin when I think that is a 2-dimensional way of looking at it. While the linear relationship between factor rank and return is a convenient approximation one must recognize its limitations.
On the other hand, the time weighted return metric you suggest I think is an idea that has merit and should be implemented. I also thought Jtbaccarat’s assertion that knowing the number of stocks per bucket was crucial although after thinking about it some more I’m now unsure. Sometimes I guess you won’t know what will be useful until you have it and have the opportunity to use it.
Hi all -
You make good points. However, as I said before, I’m not sure what Stephano’s role is in development of the tools but he seems to be the EXCEL Add-in guy. So I’m not sure how this is impacting any other feature requests.
“The biggest thing stopping P123 users from designing better systems is not a more effective analysis tool — rather it is the relatively SHORT DATA HISTORY currently available.”
It is my understanding that longer data history is in the works and will be available in the future. Now that being said, more data is going to make the process of developing ranking systems even more painful. And there will be much greater demand to build new ranking systems.
"Danparquette has already shared his studies that implemented that and I don’t see how we’re going to come up with anything significantly better going down that road. I think it leads to the problem you are stating about the same stocks coming up again and again. "
The same stocks coming up again and again is due to our inability to develop new ranking systems with higher liquidity stocks.
“I hope the developers adopt a problem solving approach to any redesign of the ranking system testing tools. Identify shortcomings and limitations of the current setup and methodology and then fix them rather than introducing new stuff indiscriminantly.”
(Marco please don’t be offended - P123 is fantastic). I’m identifying shortcomings and limitations with the current ranking menu page setup: IT SUCKS.
The whole purpose of this thread is to provide discussion so that something is not introduced indiscriminantly. Marco and Stephano have chosen to ask for people’s opinion before introducing this tool.
Steve
o806:
I think you are absolutely right. The main obstacle to developing better systems is the short period of time for which we have data.
Essentially, we are looking at a period where small cap value was king. Small wonder that it is so difficult to design good systems for large cap or, worse, large cap growth. The data are simply not there.
I continue to believe that additional data, going back deep into the 90’s, should be the best use of resources.
There are of course two points of view. One says that the more data the better. The other point of view is that recent data is more pertinent than old data. The world has had significant changes in the last ten years and we may not see the market conditions of past for a long time to come.
In any case, the existing user’s interface will impede development of new ranking systems, ten years of data or not. If I have to sit on my 3GHz pentium and do endless hours of manual ranking performance runs in the future then I probably won’t be sharing results with others. A question you might want to ask yourself is: when you get the ten years of data are you going to use ranking systems as is, develop your own new systems or do you expect others to generate systems for you?
This system (although in it’s infancy) provides some evidence that it is possible to develop largecap systems with the existing data. It was developed without use of Dan’s spreadsheet. http://www.portfolio123.com/port_summary.jsp?portid=213067
"Steve, I don’t mind at all when I’m buying and selling the same stocks hundreds of people are. As long as I’m buying before they’re buying and I’m selling before they’re selling, the more the merrier! "
Don’t forget about the tax implications of selling/ rebuying the same stocks within a certain time period.
Steve
In looking to score a ranking system, a very important side benefit would be the ability to assess the momentum of a ranking system.
To have a sense of a ranking system’s momentum, we need to give the ranking system a high score if it approximates the market fractiles (see earlier post in this thread) and a low score if it does not.
If a ranking system score is declining then it is losing its ability to forecast market performance. If a ranking system score is increasing, it is gaining its ability to forecast market performance.
We could discover and make use this phenomenon by using formulas like PrevRankScore(week) or RankScore%Chg(weeks).
But then again, why should we care about ranking system momentum? Will this momentum evaluation capability improve port performance? Who knows. But it is a major question in factor analysis research in both academia and in hedge fund research. And my hunch is that ranks do have momentum, and using that will improve performance.
If you research the factor analysis performed at Duke (Fuqua School of Business) in the Campbell Harvey investments course (google - its on line) you will find that this is a big area for future research. The work done to date shows that factor fractile (i.e. ranking systems) performance shifts - and often dramatically - this phenomenon explains why ranking systems atrophy over time and then can return to favor.
A ranking system that has its top fractile gaining at the top end of the market and its bottom fractile at the bottom of the market can suddenly have an off year, and the fractile performance characteristics can reverse! Imagine your long short portfolio where the longs are falling in price and the shorts are gaining - your “safe” hedge becomes a wipe-out! It happens…
But factor fractiles (ranks) don’t “switch” performance overnight. It is gradual. Has to be. Maybe ranks have 3 mos momentum, maybe 6, maybe 9; and maybe it can be forecasted after 1 mo, 3 mos, or 6 mos respectively. Nobody knows right now - at least nobody who knows is writing about it! But that is something we need to find out.
Wouldn’t it be cool to incorporate dynamic ranking in our Ports as the momentum of a rank changes?!
It starts with scoring the ranks. Then we need function(s) to assess rank momentum. We can manually switch to different ports in the time being, but ultimately, maybe even the buy sell rules could allow us to pick the rank range depending on the momentum scores of the rank.
This is a promising area indeed!
Carl
Steve,
We may not see another bubble for 30 years, but all of the other conditions of the past could reappear at any time. Having more data might not change my systems very much, but more data would help in several ways.
1— Longer history would increase my confidence in the system, thereby making it easier to continue through any down turn.
2— Longer history might reveal an element of risk not see in recent data and thus this could change my money management even though the ranking and buy/sell rules might remain unchanged.
3— Longer history would enable me to more precisely tell when a system has stopped working. If I just have a short test period, and then the system underperforms in the future, I have no way to tell if periodic underperformance is “normal” and it will likely start working again, or if underperformance is "abnormal’ and likely means the system is broken. Longer history tests can help like nothing else for this.
4— Longer history would help detect curve fit systems, allow for “out of sample” testing, etc.
Regards,
o806
I absolutely agree 400% with Brian. The biggest feature to add to Portfolio123 is more data.
I’m currently just trading systems that use factors that have been proven to be significant in the academic literature on time spans larger than 10 years.
I have no confidence that the amount of data that we have is enough to trust my money on. I just use this site to fine tune Sale rules and to simplify portfolio management, but I don’t much rely on it for Ranking systems.
It might be different if you are developing short term system (or not).
.luca
o806 -
I don’t disagree with having more data. I just don’t feel that more data is the prime impediment for developing new ranking systems. I think the user’s interface is the prime impediment. With the powerful PCs that most of us have it doesn’t make sense to be doing such manually repetitive tasks for hours on end. I feel like I am back in the dark ages.
Also, I’m not just talking about the tech bubble occurring in the last ten years. There are also the accounting scandals and option fixing fiascos that have left investors wary of big companies, the shift from large brokers to deep discount internet brokers, the rise of the internet as the main source of information (for example Motley Fool flogs smallcap companies). I am sure that there are many other factors I haven’t thought about.
Steve
Steve,
Yes, there are a lot of changes. In addition to the one’s you mention, there is the decimalization of trade prices and lower commissions, both of which greatly reduce slippage. Thus some high turnover systems might test with great profits pre 2000 but not be profitable today. On the other hand, if a system does not test well for pre 2000 data, that would be a very bad sign indeed.
So I am really not expecting pre 2000 data to let me find a new system. Rather I want pre 2000 data to “stress test” systems that test well over the last 5-6 years. I want to know the worst as well as the best.
This is really not an either-or issue. We need both more data and easier interfaces.
o806 -
I can agree with that. I view more data as a catalyst for new ranking systems. The current user’s interface is the impediment to new ranking systems.
It is just that some people are indicating that a metric (and excel add-in) are not necessary and there are better uses for the limited resources at hand. This is where I get my back up.
Steve