Different Results from different accounts

Report a problem or error
1

I have multiple accounts with P123 because it's a fantastic service and worth every penny and multiple accounts enable me to do more faster.

However, I've been encountering factors that yield different results from different accounts. The following ranking system contains 4 of the factors that often return different results from different accounts:

<RankingSystem RankType="Higher">
	<StockFormula Weight="0" RankType="Lower" Name="Inst#ShsOwnPQ / SharesFDPQ" Description="" Scope="Industry">
		<Formula>Inst#ShsOwnPQ / SharesFDPQ</Formula>
	</StockFormula>
	<StockFormula Weight="0" RankType="Higher" Name="DivPS(0,QTR)/DivPS(0,TTM)-FMEDIAN(&quot;DivPS(0,QTR)/Di" Description="" Scope="Universe">
		<Formula>DivPS(0,QTR)/DivPS(0,TTM)-FMEDIAN(&quot;DivPS(0,QTR)/DivPS(0,TTM)&quot;, #Industry)</Formula>
	</StockFormula>
	<StockFactor Weight="0" RankType="Higher" Scope="Sector">
		<Factor>Inst%Own</Factor>
	</StockFactor>
	<StockFactor Weight="0" RankType="Higher" Scope="Industry">
		<Factor>Inst#ShsPurch</Factor>
	</StockFactor>
</RankingSystem>

This zip file contains:

  1. the ranks for this system as of 2024-04-08 from two different accounts run within seconds of each other.
  2. results for the ranking system performance test from both accounts run within seconds of each other.
  3. screenshots of the settings for the ranking performance test from each account
  4. You can see from the ranks that the list of included tickers is the same which shows that the universes are defined identically on each account.
  5. You can see in the performance test results that the bench and universe returns are identical, but the returns of the bins differ throughout the period, which shows both that the differing rank data affects the simulation engine and that the rank data differs throughout the historical dataset.

This is NOT a widespread problem. There are only a dozen or so factors/formulas out of hundreds that I've tested that return inconsistent results from different accounts. DivPS, ULTOSC, and the Institutional Ownership factors are the primary factors where I see the errors.

2

Still investigating; will need to dig deeper.

Is it consistent? Same differences all the time?

3

No, the results from the same account don't appear to be consistent. I just pulled the ranks for 4/8 again from account 2 and they're as different from what was returned for that account yesterday as the ranks from account 1. Super weird.

4

We're still investigating. We have been able to reproduce different results. Sorry it's taking so long

1 Like
5

We think it's been corrected. There were two distinct problems both due to non deterministic behavior only noticeable using different accounts, or using the same account after at least 30 min have passed (the duration of our rank cache).

The good news is that it only affects a small subset of the stocks and only DivPS function and Inst factors. The bad news is that when these factors are used in ranking systems they can have ripple effects and change rank orders somewhat. The more factors you have the less noticeable the differences.

For the Inst factors (only in North America) some companies were pulling in data from two different issues of the company, for example different share class. It's not something we expected therefore which data was chose was random. Only a small number of companies were affected directly with minor ripple effects in ranking of others in the universe.

For DivPS function there were a handful of stocks that had dividend data but no fundamentals, which is quite strange. This caused an unhandled situation with nondeterministic behavior in their ranks which had a minor ripple effect to the ranks of others in the universe.

Lastly, we don't think ULTOSC has any issues.

Let us know if you still see the problem. Very sorry about this, and thank you for finding it.

6

Thanks. DivPS and Inst are now working on my side.

I'm still having issues with ULTOSC. But looking closer it may be unlike the other two. Using this system:

<RankingSystem RankType="Higher">
	<StockFormula Weight="0" RankType="Higher" Name="ULTOSC(0)" Description="" Scope="Industry">
		<Formula>ULTOSC(0)</Formula>
	</StockFormula>
</RankingSystem>

When I pull the latest ranks (or any given date's ranks) from different accounts, the stocks that tie for the factor are often sorted differently within the tie. So depending on how the tie is broken, different stocks end up in different rank positions (and also different buckets in a rank performance test). I hadn't noticed that it was only the ties that were being sorted differently before.

7

There's no "tie-breaking" logic. The order of stocks with the same score should be considered meaningless and can only be fixed using other factors.

Having said that what stocks are producing exactly the same ULTOSC ? Seems highly improbable.

8

P123_Rankings_355635.csv (370.0 KB)

A surprising number of ties. I pulled two ranks with identical settings into that file to compare them.

The rank position order of tied stocks using other factors is stable. Only reason ULTOSC struck me as odd.

9

I see . You are ranking vs the Industry. That will generate many ties. The order of those ties is meaningless and nondeterminstic, so you need additional factors to break the ties.

10

Looking closer at the rank data in my last post, there are differences beyond order of ties. For example, ABBV has a stock rank of 37.65 in one run and 37.06 in the other. ACHC has a rank of 54.59 in one and 52.27 in the other. It is extremely rare to have two consecutive runs agree on the ranks for ULTOSC, even after accounting for the ordering of ties.

11

Sorry for the delay. We did find a problem with ULTOSC. Stocks with less than 100 bars at the time of the evaluation results in a non-deterministic behavior, which then affects the relative rank of others. We'll have a release later today.

Thanks again

12

Thanks for being on top of all of this Marco and team. The amount of work required to build and maintain the point in time database seems unending, but is very appreciated.