Skip to main content

Development Setup

This guide helps you set up a development environment for working with Merlin, whether you're extending its capabilities or simply running it locally.

Prerequisites​

Before starting, ensure you have:

  • Python 3.11+ installed
  • Git

Installation​

Clone the Repository​

Start by cloning the Merlin repository:

git clone <repository-url>
cd merlin

Create a Virtual Environment​

Using a dedicated virtual environment is recommended:

# Create a virtual environment
python -m venv venv

# Activate the virtual environment
# On Windows:
venv\Scripts\activate
# On macOS/Linux:
source venv/bin/activate

Install Dependencies​

Install the required dependencies:

# Install development dependencies
pip install -r requirements-dev.txt

# Install the package in development mode
pip install -e .

Configuration​

Environment Variables​

Set the following required environment variables:

# MesoSim and Q-API configuration
export MESOSIM_INSTANCE="https://portal.example-customer.mesosim.io"
export Q_API_INSTANCE="https://q-api.example-customer.mesosim.io"
export MESOSIM_API_KEY="your-api-key"

# Control parallel execution
export BACKTESTS_IN_FLIGHT=10
export MODEL_CALLS_IN_FLIGHT=10

For Windows users, use set instead of export.

Configuration Files​

Merlin uses JSON configuration files for:

  1. Strategy Definitions: Define the trading strategy parameters (see boxcar-strategy.json)
  2. Strategy Model Configuration: Define optimization parameters (see boxcar-model-cfg.json)
  3. Portfolio Configuration: Define portfolio optimization parameters (see portfolio_no_strategy_opt.json)

Sample configuration files are available in the configs/ directory.

API Documentation​

Merlin interacts with two main APIs:

  1. MesoSim API: For running option strategy backtests
  2. Q-API: For statistical analysis, model building, and optimization

For comprehensive API documentation, refer to the Q-API documentation:

Extending Merlin​

Adding New Features​

To add new features to the feature engineering process:

  1. Modify feature_engineering.py to include your new feature calculations
  2. Update any related documentation
  3. Add tests for the new features

Example extension to add a new feature:

def calculate_custom_feature(df):
    """Calculate a custom feature from dataframe data."""
    df['my_custom_feature'] = df['some_column'] / df['another_column']
    return df

# In the extract_features function:
if add_custom_features:
    df = calculate_custom_feature(df)

Troubleshooting​

Common Issues​

  1. API Connection Errors

    • Check that environment variables are set correctly
    • Verify API key permissions
    • Confirm network connectivity to API endpoints

  2. Dependencies Issues

    • If you encounter dependency errors, try updating with: 
      pip install -r requirements-dev.txt --upgrade

  3. Parallel Execution Issues

    • If you experience timeout or resource limitation issues, reduce the values of BACKTESTS_IN_FLIGHT and MODEL_CALLS_IN_FLIGHT

Logging​

Merlin uses Python's standard logging module. To enable detailed logging:

import logging
logging.basicConfig(level=logging.DEBUG)

All Merlin operations create a log file in the merlin-results directory.