Modeling Considerations

Effective modeling with Merlin requires understanding the statistical challenges inherent in financial data, selecting appropriate features, and implementing proper validation methodologies. By considering these factors and applying the principles outlined in this guide, you can develop more robust predictive models that enhance trading strategy performance across varying market conditions.

Market Dynamics and Feature Selection

Adaptive Feature Relevance

Optimal predictive models utilize specific, meaningful features with logical relationships to PnL outcomes. However, financial markets exhibit non-stationary behavior across time:

• A predictor showing high Cramér's V correlation during Period A may display diminished predictive power during Period B
• Relationships between features and targets often decay when evaluated across extended timeframes
• Market regimes shift, causing previously reliable feature-target relationships to change or invert

This temporal instability makes comprehensive manual pre-screening of predictors impractical. Merlin's Stepwise ensemble approach addresses this challenge by automatically selecting relevant features for each walk-forward fold, adapting to changing market conditions through time-localized selection.

While Merlin's Stepwise algorithm can process a large feature set, its effectiveness increases significantly when provided with a curated subset of strong candidate predictors rather than an exhaustive collection. The optimal approach combines domain expertise with statistical pre-screening:

1. Identify a focused set of 10-20 features with theoretical justification for their predictive relationship
2. Validate these candidates through preliminary statistical testing (e.g., Cramér's V analysis)
3. Configure Merlin to apply Stepwise selection within this pre-screened feature pool

This focused approach improves model interpretability, reduces overfitting risk, and typically yields more stable models than an unconstrained "all-in" feature selection strategy. The RegressionModelConfigs parameter in Merlin provides precise control over candidate features for each model within the ensemble, allowing you to implement this targeted selection approach.

Feature Stability Assessment

When developing trading models, consider:

• Feature persistence: How consistently does a feature maintain its predictive power across different market regimes?
• Feature degradation rate: How quickly does the feature-target relationship decay over time?
• Feature interaction effects: Do combinations of features exhibit emergent predictive properties beyond their individual contributions?

Statistical Properties of Trading Data

Stationarity and Independence Challenges

Statistical models perform optimally when applied to data that exhibits:

• Stationarity: The statistical properties (mean, variance, autocorrelation) remain constant over time
• Independence: Each observation is unaffected by previous observations
• Identical distribution: Observations are drawn from the same probability distribution

In financial markets, these conditions are inherently violated:

• Non-stationarity: Markets evolve through different regimes, exhibiting changing statistical properties
• Serial dependence: Trading outcomes exhibit significant autocorrelation, particularly in strategies with longer holding periods
• Heterogeneous distributions: Market behaviors cluster into distinct regimes with different underlying distributions

For enhanced performance, users can extend Merlin with additional techniques:

1. Feature normalization: Implementing de-meaning and centering to induce pseudo-stationarity in features
2. Differencing transformations: Converting non-stationary series into stationary ones by taking differences
3. Statistical transformation: Applying Box-Cox or similar transformations to stabilize variance

Please refer to the Development Setup page for more details on how to get started extending Merlin with these techniques.

PnL Series Transformation Methodology

Merlin implements a transformation process to convert raw trading data into a modeling-compatible format:

1. Position identification: The system identifies Exit Position Events and traces back to corresponding Entry Position Events
2. Feature state capture: The state of all relevant features is recorded at entry time
3. Terminal PnL extraction: The final profit/loss outcome for each position is captured
4. Time-series construction: A structured time series is created where:
   • The timestamp corresponds to position entry time
   • The target value is the position's terminal PnL
5. Cumulative aggregation: A daily time series is constructed by cumulating Position PnLs

This transformation creates a PnL curve equivalent to a strategy with one-day holding periods, which offers several statistical advantages:

• Reduced autocorrelation: Minimizes serial dependence between sequential observations
• Cleaner signal extraction: Removes intra-position noise from the modeling target
• Standardized comparison framework: Enables consistent evaluation across strategies with different holding periods

For strategies with multi-day holding periods, this approach compresses multiple days of price action into a single entry-day data point. While this represents a dimensionality reduction, it preserves the overall return profile. To evaluate this transformation's impact, Merlin provides the --include-mesosim-wf-base parameter, which executes a MesoSim backtest on the same out-of-sample period as the walkforward. This allows direct comparison between the wf-base and the wf-oos-model results via the generated tearsheets.

Prediction Horizon Considerations

The accuracy and reliability of predictive models typically degrade as the forecast horizon extends:

• Short-term predictions (1-5 day holding periods) generally exhibit higher accuracy due to:

  • Stronger signal-to-noise ratio in short-term market behaviors
  • Fewer exogenous events that could disrupt predicted patterns
  • Lower parameter uncertainty in shorter forecast windows

• Medium- to Long-term predictions face increasing challenges:

  • Higher exposure to unexpected market events
  • Compounding of model error terms
  • Regime shifts becoming more probable
  • Multiple regime changes may occur within the forecast window
  • Accumulation of stochastic market noise overwhelms signal

This predictive horizon limitation is one reason why systematic models often perform more consistently on strategies with shorter holding periods.

Strategy Characteristics and Model Efficacy

Entry Filtering Effectiveness

Entry filtering methodology exhibits varying effectiveness based on strategy characteristics:

• Highly effective for static strategies: Positions that maintain consistent exposure profiles throughout their lifecycle benefit most from entry filtering
• Moderately effective for fixed-adjustment strategies: Predetermined adjustment schedules can be incorporated into the entry model
• Less effective for dynamic-adjustment strategies: Strategies that adjust positions based on evolving market conditions introduce variables unavailable at entry time

Strategies with minimal reliance on P&L derived from post-entry adjustments provide more predictable modeling targets. The performance gain from entry modeling tends to be inversely proportional to the strategy's dependency on dynamic adjustments.

Potential for Dynamic Exit Modeling

While Merlin currently implements Entry Filtering, the SETS model could theoretically be applied to develop dynamic exit signals. This approach would be particularly beneficial for strategies with longer holding periods, where market conditions may change substantially between entry and exit.

This type of extension would enable the model to reevaluate positions based on evolving market conditions and potentially signal early exits when the probability of profitable continuation decreases or the chances of market crash is increasing. Though not currently implemented in Merlin, this represents a potential future enhancement to optimize performance for longer-dated trades.

Feature Design Principles

Optimal feature engineering for Merlin follows these guidelines:

• Continuous over categorical: Features that output continuous values across a range provide richer information content than discrete categorical outputs
• Full range utilization: Effective features should utilize the full spectrum of possible values rather than clustering at boundaries
• Avoid binary/ternary outputs: Features that output only discrete values like -1, 0, 1 limit the model's ability to capture nuanced relationships
• Information density: Each feature should provide unique, non-redundant information about the target variable
• Domain relevance: Features should have theoretical or empirical justification for their relationship to trading outcomes

Traditional technical indicators can also serve as features when properly transformed to meet these criteria.

Advanced Feature Selection Methods

While Merlin implements Stepwise feature selection by default, additional feature pre-filtering approaches can enhance model performance:

• Correlation analysis: Identifying and eliminating redundant features that capture similar market dynamics
• Variance thresholding: Removing low-information features with minimal variation
• Principal component analysis: Creating orthogonal feature combinations that capture maximum variance

These techniques can be implemented as pre-processing steps before applying Merlin's main modeling workflow. For customized feature selection pipelines, contact support to discuss implementation options.

Implementation Considerations

Realistic Expectations

Model development in quantitative trading requires tempering expectations:

• The effort required to develop effective predictive models often rivals the effort of strategy development itself
• Not all strategies benefit equally from predictive modeling
• No model can consistently predict all market conditions

Deployment Cycle

For optimal results, implement a systematic deployment process:

1. Regularly retrain portfolios (e.g., weekly) to obtain updated Strategy Inclusion lists
2. Within each period (e.g., daily), run the Strategy Optimizer with Last Fold Enabled to generate current Optimized Trade Plans
3. Monitor model performance metrics to detect potential degradation
4. Maintain a comparison benchmark of unfiltered strategy performance