AI in investment · model drift after deployment
Nebula Pilot Research — when an investment model keeps moving after it goes live
We publish free editorial and informational articles on how investment models behave once they leave the lab, and how teams read the drift that follows. Nothing is sold here; inquiries only.
The problem we keep seeing
A model that cleared backtesting can still lose its footing in the market it was built for — regimes shift, features drift, and the gap between training data and live conditions widens a little every week.
What our editorial work is aboutHow we read drift after deployment
Four checkpoints for a model that has already left the notebook
Our articles walk through evaluation as a sequence, not a single score. Each step answers a question that surfaces only after a model starts trading live capital or live signals.
Establish the post-deployment baseline
Fix the window, the feature set, and the target definition the model was approved on, then snapshot its live performance against that reference rather than against a retrained version of itself.
Separate data drift from concept drift
Track population stability index and Kolmogorov–Smirnov statistics on the input distribution first; only when the inputs are stable do we read a drop in prediction error as a sign that the input-to-output relationship itself has moved.
Decompose the error, not just the loss
Break a degraded metric into slice-level contributions — sector, horizon, volatility regime — so a model that looks fine on average is not hiding a single segment where it has quietly broken down.
Decide between recalibration and retirement
Set the thresholds in advance: which drift magnitude triggers a reweight, which triggers a full retrain, and which pulls the model out of production until the cause is understood.
Inquiries
Ask us about a model that has started to drift
We answer editorial and research questions about evaluating model drift after deployment. We do not sell advisory engagements, manage capital, or give personalised investment advice.
Who this is for — and not for
This is for practitioners who already run a model in production and need to argue, with evidence, whether it should stay there. It is not a signal service, a managed account, or a place to buy investment advice.
Editorial scope of Nebula Pilot ResearchQuestions we hear
Four objections to evaluating drift this way
"Backtesting was already strong — why keep checking?"
Because a backtest is a statement about a past distribution, and the live distribution is the one paying the cost. A model can pass every historical fold and still decay within a quarter once the regime that produced those folds ends.
"Is drift the same as the model being wrong?"
Not always. Inputs can drift while the learned relationship still holds, in which case the fix is a data refresh, not a retrain. We separate those cases before recommending any change to the model itself.
"Do you retrain the model for us?"
No. We write and share editorial articles about how to evaluate drift; we do not build, host, retrain, or operate models on anyone's behalf, and we do not place trades or hold client assets.
"Is any of this investment advice?"
No. Nothing on this site is personalised advice or a recommendation to buy, sell, or hold any instrument. Our articles describe evaluation practice; what you do with a model is your decision.