How We Test Strategy Credibility

The foundation of honest backtesting is data isolation. We split every ticker's historical data into three non-overlapping segments:

• Train (50%) — the optimizer explores parameter combinations here.
• Validate (25%) — used to select the best candidate from the training results.
• Holdback (1-year minimum) — data the strategy NEVER sees during optimization. Holdback returns are the closest proxy for live performance.

Within the training period, we run a 5-window rolling train/test protocol. Each window trains on a portion of the data and tests on the immediately following out-of-sample segment.

• Strategies that fail walk-forward are eliminated automatically before reaching the validation phase.
• This tests whether the strategy adapts to shifting market regimes — trending, mean-reverting, and volatile environments.
• Only strategies that perform consistently across all five windows advance.

Pardo, R. (2008). "The Evaluation and Optimization of Trading Strategies." Wiley.

When an optimizer tests hundreds or thousands of parameter combinations, the best result will look impressive by pure chance. The Deflated Sharpe Ratio (DSR) corrects for this multiple-testing bias.

• Uses the number of independent trials to adjust the significance threshold for the Sharpe Ratio.
• Accounts for non-normality (skewness, kurtosis) of return distributions.
• A Sharpe of 2.0 from 1,000 trials is statistically very different from a Sharpe of 2.0 from a single trial. DSR quantifies this difference.

Bailey, D.H. & López de Prado, M. (2014). "The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality." Journal of Portfolio Management, 40(5), 94–107.

We use a permutation test to determine whether the strategy's returns are statistically distinguishable from random chance.

• Shuffles daily portfolio returns using block bootstrap to destroy the timing signal while preserving serial correlation structure.
• Runs 1,000+ permutations per test to build a null distribution.
• The strategy must achieve a p-value below 0.05 — meaning there is less than a 5% probability that random reordering would produce equal or better returns.

White, H. (2000). "A Reality Check for Data Snooping." Econometrica, 68(5), 1097–1126.

A strategy that looks profitable with zero transaction costs often is not. We apply broker-specific commission schedules to every backtest.

• 12 broker presets: IBKR Pro Tiered, IBKR Pro Fixed, Robinhood, Schwab, Fidelity, E*TRADE, TD Ameritrade, Webull, Firstrade, TradeStation, Tradier, and Alpaca.
• Per-share rates with minimum and maximum per-order caps.
• SEC Section 31 fees and FINRA TAF regulatory fees applied to every sell order.

After optimization selects the best parameters, we perturb each one to see if performance degrades gracefully or collapses.

• Each optimized parameter is varied ±20% from its selected value.
• If performance collapses outside a narrow range, the strategy is flagged as curve-fitted.
• Robust strategies exhibit a performance plateau — they work across a range of nearby parameter values, not just one magic setting.

Every signal and strategy operates within a market regime — bull, bear, sideways, or crisis. alphactor.ai classifies the current regime using a three-layer ensemble grounded in two of the most cited papers in financial econometrics.

• Layer 1: Hamilton (1989) Markov-Switching Regression fits a 2-regime model on log returns with switching variance. The Hamilton smoother produces proper Bayesian posterior state probabilities — not ad-hoc threshold rules.
• Layer 2: Kritzman et al. (2012) volatility overlay detects crisis conditions by ranking current realized volatility against its full historical distribution. Bear markets with volatility above the 80th percentile are escalated to Bear/Crisis.
• Layer 3: SMA-50 trend confirmation prevents false bull signals during bear-market bounces and vice versa, providing the Sideways classification when the MS model is uncertain.

Hamilton, J.D. (1989). Econometrica, 57(2). · Kritzman, M., Page, S., & Turkington, D. (2012). Financial Analysts Journal, 68(3). · Nystrup, P., Lindström, E., Pinson, P., & Madsen, H. (2024). "Learning Hidden Markov Models for Regression with Unaligned Timestamps." arXiv:2402.05272.

Knowing the current market regime is only half the problem. The other half is selecting which strategy to deploy in each regime. alphactor.ai uses two regime-conditioned selection methods inspired by mixture-of-experts architectures in machine learning.

• Hard Switch — a single champion strategy is selected per regime via argmax over regime-conditional expected scores. Hysteresis and a switching penalty prevent whipsawing between strategies during regime transitions.
• Soft Mixture-of-Experts (MoE) — instead of picking one winner, a softmax gate blends the top-K strategies weighted by their regime-conditional scores. A hierarchical family-level gate groups strategies by signal type, preventing overconcentration in one signal family.
• Both methods use an online regime posterior (the Hamilton filter from Layer 7) so that strategy weights at each bar depend only on data available at that point — no lookahead bias.

Meta-Learning Mixture of Experts for Regime-Aware Portfolio Construction. arXiv:2505.03659.

Picking which strategy to surface as the champion is itself a statistical decision. We rank candidates with a hard credibility floor + cup-tier priority — so a backtest only earns the champion slot if it both beats the underlying stock on the held-out year AND survives our credibility tests.

• BEATS cup tier (1y holdback) — strategies are graded 🏆 to 🏆🏆🏆🏆 based on whether they beat the underlying stock and SPY on the held-out 1-year window. cup_4 = beats stock and SPY both by ≥5pp; trails = doesn't beat the stock.
• Hard credibility floor — a candidate must pass p-value ≤ 0.05 AND DSR ≥ 0.20 AND ≥ 30 trades to be eligible for cup-priority promotion. Below the floor we fall back to the credibility-haircut composite score, so a credible-but-trailing champion never gets displaced by a noisy cup_3 from too few trades.
• Cup priority within the credible cohort — once the floor is met, cup_4 outranks cup_3 outranks cup_2, with composite score as the tiebreak. This stops the system from rewarding a marginal in-sample edge over a strategy that actually beat the stock on the held-out year.

When a strategy can go short, every percentage return, drawdown, and Sharpe must be computed with the correct sign — or the cup tier shows a strategy that looks profitable but isn't. Phase 1 ships an internal accounting engine that handles long, short, and mixed books with textbook-orthodox math.

• Short proceeds credit cash at the open; margin reserved at Reg T 50% initial, 30% maintenance. Buying power equals equity minus margin used, never "available cash".
• Daily borrow accrual: short_qty × close × annual_borrow_rate / 252, debited from cash each bar. Backtests assume 2%/yr default; hard-to-borrow names bump to 15%/yr.
• Forced cover trigger: when equity drops below 30% of short market value, the largest-loser short closes at the next bar — same behavior your real broker would force on you.

See docs/short-selling-plan.md §7.2 and §10 for the accounting equations and the 20-test regression corpus that gates v2 from breaking long-only behavior.