# Monte Carlo permutation test

How is a p-value computed when a trading signal is tested against shuffled returns?

Count the shuffled datasets whose statistic is at least as large as the observed one, add one, and divide by the number of shuffles plus one. With 99 shuffles and 4 such values, p = (4 + 1) / (99 + 1) = 0.05.

The p-value holds when shuffling respects the null hypothesis; serially dependent returns need block or restricted permutations.

A permutation test compares an observed signal statistic with statistics recomputed after a specified association is broken under a null hypothesis.

## The permutation target

For a signal test, shuffle the alignment between signals and subsequent returns, then rerun the full statistic. Permuting returns independently also changes their time dependence; block or restricted permutations retain chosen structure. The exchangeability assumption must match the null being tested. Ojala and Garriga distinguish label permutations from restricted feature permutations in classifier tests.

## A finite enumeration

Suppose four fixed signal exposures are 1, 1, 0, 0 and four returns are 3, 2, -1, -2 percentage points. The observed sum for exposed periods is 5 points. Across the six ways to select two of four returns, the sums are 5, 2, 1, 1, 0 and -3 points. One of six is at least 5, giving an exact one-sided p-value of 1/6, about 0.167. This small example assumes exchangeable returns.

```
Monte Carlo p = (1 + permutations with statistic >= observed) / (B + 1)
```

## Sampled permutations

When the permutation space is large, draw B valid permutations and use the plus-one p-value shown above. With 99 draws and four statistics at least as large as observed, p = (4 + 1) / (99 + 1) = 0.05. State the statistic, null, restriction, draw count and random seed. If many signals were screened, rerun that selection within each draw or adjust the family of tests.

## Different from path simulation

A Monte Carlo equity-path simulation rearranges or resamples returns to study a distribution of drawdowns or outcomes. A permutation test uses a null distribution to assess a predeclared statistic. Stochastly's ML family computes label-permutation negative controls. Its PBO permutation p-value node is marked roadmap; a control AUC is not the p-value calculated above.

## Frequently asked questions

### What does a small permutation p-value mean?

Under the specified exchangeability null, few shuffled datasets produce a statistic at least as large as the observed one.

### Can I shuffle individual daily returns?

That requires exchangeability at the daily level under the stated null. Serial dependence may require block or other restricted permutations.

### Is this the same as Monte Carlo drawdown simulation?

No. A drawdown simulation estimates path outcomes; a permutation p-value compares a statistic with a null distribution.

## Sources

Ojala and Garriga (2010). [Permutation Tests for Studying Classifier Performance](https://www.jmlr.org/papers/v11/ojala10a.html). Journal of Machine Learning Research 11, 1833-1863.

North, Curtis and Sham (2002). [A Note on the Calculation of Empirical P Values from Monte Carlo Procedures](https://doi.org/10.1086/341527). American Journal of Human Genetics 71(2), 439-441.

White (2000). [A Reality Check for Data Snooping](https://doi.org/10.1111/1468-0262.00152). Econometrica 68(5), 1097-1126.

[Primary source for Monte Carlo permutation test](https://doi.org/10.1086/341527)

## In the library

Shift and permutation controls for autocorrelated features

[Variable Importance Is Biased, and Permutation Extrapolates](https://stochastly.com/library/machine-learning/variable-importance-is-biased-and-extrapolates)

## Related

[Monte Carlo simulation for trading](https://stochastly.com/learn/monte-carlo-simulation-trading)

[Multiple testing in trading](https://stochastly.com/learn/multiple-testing-in-trading)

Permutation and shift controls

---
Source: https://stochastly.com/learn/monte-carlo-permutation-test