When I have used Time Series CV, does it matter much if you choose to have a separate holdout at the very end? I understand the point of a holdout is to ensure that you run the test on a period that is completely unseen by the model, but with Time Series CV and a 12-month holdout and a 30-week gap, isn't that the same as if you had a holdout at the end of the period? Each holdout period during a Time Series CV should function similarly as if it were a holdout at the end of the period.
Regarding the pricing of the AI factor itself: If I want to use this model as I usually do for ranking and manual weekly screening, I have trained the model according to the pricing for "Training and Validation." However, do I actually need this prediction cost (100 dollar) ? Is it not possible to just run the prediction under "predict" once a week up to the latest date? (Wouldn't that yield the same result?)
Here is the English translation of your Norwegian text, adhering to all the specified guidelines:
"Here is my third question. I tried to test a strategy set up by P123, but with a few minor adjustments, and I'm getting extremely different results.
It uses the same features, but the only changes were to "Max Return" and altering the universe to "Russell 1000." I also used "Time Series CV" as the validation model. Otherwise, there were no changes, yet the returns are drastically different. My minor adjustments led to an enormous change from a 32% return to 7%, becoming extremely unstable. Why is this the case?"
It cannot be the universe (both large cap), nor can I understand that the Validation Method should yield such results. Could it be due to the Max Return setting?
Generate fresh predictions up to the current date.
Use those predictions in a manual ranking or screening process.
Manually manage buys, sells, position sizing, and record keeping.
Then, in principle, you can obtain the same prediction values by running the prediction process yourself each week.
The tradeoff is that everything after the prediction step becomes manual. The platform is no longer maintaining a live strategy for you, tracking historical signals, generating recommendations, applying buy rules, applying sell rules, calculating turnover, monitoring performance, or maintaining the statistics that come with a strategy.
So from a pure signal-generation perspective, running predictions manually each week can produce the same ranking inputs. The prediction service becomes valuable when you want the platform to continuously maintain and operationalize those predictions as part of a live investment process.
If you're only running a single model and are comfortable with a weekly manual workflow, you may find that generating predictions yourself is sufficient. If you're managing multiple AI models or strategies, the automation and tracking provided by the prediction service tends to save a significant amount of time and reduces the risk of operational errors.
The changes you made aren't minor and each pulls a different way, so the drop is confounded.
Time Series CV changes what you're measuring. A single holdout scores one window, which can be a lucky draw but can also be optimized to the current regime. CV averages X walk-forward windows, and the early folds train on fewer years, so under-trained folds drag the mean down and create the instability.
S&P 500 to R1000 swaps in ~500 mid-caps, which is a different factor exposure and much wider dispersion, not just more tickers. Max Return then amplifies it.
Best way to test things: change one lever at a time from the original, and compare accordingly.
