It's only available in the screener right now. We are still tweaking its RAG data and instructions. Tool wise it's very limited right now, just a proof of concept. It only knows your name, and can search through P123 screens. It's strengths are definitely explaining rules. It got a lot better yesterday writing rules because we made its instructions much bigger to make it stop using non-P123 syntax. We're still exploring alternative in this area.
We are also working on
Adding RAG data including examples, FAQs, site how-tos
Adding forum content (last 5y for ex)
Adding agentic tools
Improving UI/UX
Let us know what you think, specially what tools you'd like it to have and mistakes it makes. Also interested in hearing in how we can all collaboratively make it smarter.
Will try it out! One way to reduce hallucinations and force it to use the right syntax would be having a mixture of differently trained models and having the final arbitrator model go with the consensus answer, discarding any outlier answer. Currently LLM are bad at not hallucinating when they don't know the answer but this greatly reduces that negative aspect. Some (non) hallucination metrics:
That previous post is specifically what they do when they don’t know -which is a niche read. The ratings change over time but rather than it telling me to go with a specific model it tells me to not go with a single/unique model! Even 22% is too much for me. Of course it does not mean 22% hallucinations since its just when the model somehow does not know the answers or makes assumptions. Models know more and more these days and by having them check each other it can get quite accurate.
I don't know if the latest Open AI models are better, but I found them to be completely hopeless compared to the other major models. They tend to go in to full hallucination mode when context is to large, just have to delete the full chat to not continue to repeat it in the next chat.
If we get the documentation you have used it will be easier for us to help making it better.
I tried a bit and it is really good at explaining formulas but it struggles to create more advanced formulas.
I’m impressed. It reasonably labels concepts implied by the rules and picks up multi-rule blocks that build a concept. It pointed out and labeled some regime change and market condition logic correctly.
One thing I noted is that when the Explain Rules action has not been performed for a screen it just brings up the prior result without any notice that the explain action has not been performed for that screen. Adding a message stating that would save some confusion time.
Expanding the summary to include the impact of the interaction of the Ranking System (selection rules) would be very powerful as it makes a difference which way the selection rules are tilted (Growth, Value, …).
We have a flexible backend that can easily use any model. We could even let the user decide. But I've never seen any site that allows the user to choose the model for the agent. I wonder why.
We're still working on this , but yes, we will have it all available soon for download.
Good idea for a "tool" that allows the LLM to read the ranking system. All we need to do is add to the prompt which ranking system the screen uses and the LLM can then use the tool to read it. Thanks
Community intelligence?
The other focus are right now is to create RAG data of examples and Q/A. It's very time consuming for us, and not diverse enough. It might be worth setting up an easy way for anyone to contribute Q/A, and examples. All we would need to do is quickly review submissions before adding them to the RAG
Since we seem to have a “strong explainer / weak creator” , another option is to leverage the current ability to explain rules to create the RAG data. Perhaps also cherry pick from public screens for diversity. A quick research seems to indicate this is a good strategy ChatGPT - RAG Use of Explanations
I have tested it on a few different things, and I've noticed that I get better and more precise answers than if I use the large language models. That's great!
One thing I would like is the option to have a larger chat area, preferably one that covers the entire screen. Some of these conversations can be long, and it's a bit frustrating having such a small reading area on the right side of the webpage. So, the option to maximize the reading area for the chat.
Seems like a good idea. This seemed similar to what Alpha Go did: kept playing itself until it got good at Go. Turns out others have considered using this method for generating RAG data:
I think this last is open source. And while it uses Llama, it can be used to train other LLMs I believe. I doubt that it is the only option.
I think you could do an internal syntax check, like testing the suggested formula in a mini screen to see that you don't get an error answer back, before the LLM gives an answer. It's better to get an answer back from the Alphanaut that he failed to write the formula rather than making up a formula that does not work.
I think it will be necessary to add dedicated lookup document for every function/factor that declares its context validity. I can see that most of the information is in the reference documentation, but organizing the function/factor by context will definitely be helpful for a LLM. If you use json RAG, the context layer for each formula could look like this.
{
"type": "function_context_map",
"function": "FRank",
"contexts": {
"screen": {
"valid": true,
"usage": "filter",
"example": "FRank("EPS%ChgTTM") > 90",
"notes": "Returns percentile rank, use with comparison operator"
},
"ranking_system": {
"valid": true,
"usage": "factor",
"example": "FRank("EPS%ChgTTM")",
"notes": "Already returns 0-100, no need to normalize further"
},
"universe": {
"valid": true,
"usage": "filter",
"example": "FRank("EPS%ChgTTM") > 80"
},
"ml_feature": {
"valid": false,
"notes": "FRank is already a rank — redundant to re-rank in ML. Use raw factor instead."
}
}
}
Good idea. Rather than have us try to use the AI suggested code in a rank performance test and find the syntax is wrong, the AI could ensure the syntax has already been checked. It is as if the LLM is now agentic. It could run the formula through P123's rank performance test syntax checker behind the scenes. This is called Sandboxing.
But it could also generate code, check it, and use the successful formulas to train itself. This is what Google does with AlphaCode. So, unlike AlphaGo, it does not need to play against itself to learn—it can just run its ideas through the syntax checker.
The key is you are combining two powerful concepts: making it agentic within a single query's execution (Algoman’s idea), and using reinforcement learning for long-term training (as is done with AlphaCode).
Much better would be to open up the platform via better APIs to allow AI agents to interact with the data in a secure environment and more flexibly create and validate models.
