Is the Signal Different from the Baseline? Compare Forward-Return Distributions with Python

Histogram comparing five-bar forward returns after SMA20 Cross Above signals with all eligible AAPL bars — Figure 1. Five-bar forward-return distributions for SMA20 Cross Above events and all eligible AAPL bars. Each histogram is normalized as a percentage of its own group.

In the previous lesson, we measured what happened after an SMA crossover.

Cross Above
↓
1-bar return
5-bar return
20-bar return
↓
compare with baseline

That gave us useful summary numbers.

But a mean is only one description of a collection of outcomes.

Two groups can have similar means while having very different distributions.

Phase 4-03 asks:

Is the signal group
actually distributed differently
from ordinary market outcomes?

1. Freeze One Horizon Before Comparing Distributions

Phase 4-02 calculated several horizons.

1 bar
5 bars
20 bars

In this lesson, we freeze:

forward_horizon = 5

This is not because five bars are universally better.

We freeze one horizon so that only one new question changes:

What does the
distribution look like?

2. Keep the Signal Definition Frozen Too

We reuse the same event from Phase 4-01.

Indicator
SMA 20

Entry Event
previous Close <= previous SMA20
AND
current Close > current SMA20

Signal Time
after bar t Close

Execution-Aligned Outcome
Open(t+1)
→ Open(t+6)

Five bars are measured from the planned next-Open entry point.

3. Define the Two Groups

The signal group contains only eligible Cross Above events.

Signal Group
=
SMA20 Cross Above bars

The baseline group contains every bar for which the same five-bar forward return can be calculated.

Baseline Group
=
All Eligible Bars

This is an unconditional baseline. It asks what normally happened in the same market and period without conditioning on the crossover.

4. Use Exactly the Same Return Formula for Both Groups

A fair comparison must keep the outcome definition identical.

Entry
Open(t+1)

Exit
Open(t+6)

Return
=
Exit Open
---------
Entry Open
- 1

The only difference between the groups is whether a Cross Above event occurred.

5. Why the Mean Is Not Enough

Imagine two small groups.

Group A

-1%
 0%
+1%
+2%
+3%

Mean = +1%


Group B

-10%
 0%
 0%
 0%
+15%

Mean = +1%

The means are identical.

But the experience represented by the two groups is clearly different.

Group B has much more extreme outcomes.

6. Start with Sample Count

Before interpreting any percentage, ask how many observations created it.

Count
=
number of valid outcomes

A Cross Above event usually occurs much less often than an ordinary eligible bar.

So the signal group will normally be much smaller than the baseline group.

7. Mean Answers One Question

Mean
=
sum of returns
/
number of returns

It answers:

What was the average
historical outcome?

But a few large observations can move the mean considerably.

8. Median Answers a Different Question

Sort the observations.

The median describes the middle.

Median
→ middle observation
  after sorting

If the mean and median tell very different stories, extreme values may be important.

9. Positive Rate Answers Another Question

Positive Rate
=
number of returns > 0
---------------------
number of valid returns

A group can have:

high positive rate
but small average gains

or

lower positive rate
but a few large gains

Therefore positive rate should not replace the rest of the distribution.

10. Minimum and Maximum Reveal the Extremes

Minimum
→ worst observed outcome

Maximum
→ best observed outcome

These values can help explain why the mean moved.

But they should not be treated as expected future limits.

11. Compare Differences, Not Only Separate Numbers

The program calculates:

Mean Difference
=
Signal Mean
-
Baseline Mean

Median Difference
=
Signal Median
-
Baseline Median

Positive Rate Difference
=
Signal Positive Rate
-
Baseline Positive Rate

A difference is easier to interpret than looking at two disconnected tables.

But it is still descriptive.

12. A Positive Difference Is Not Yet Proof of Edge

Suppose:

Signal Mean
= +1.2%

Baseline Mean
= +0.8%

Difference
= +0.4%

We may report:

Observed historical
mean difference
=
+0.4 percentage points

We may not yet conclude:

predictive edge proven

statistically significant

robust

profitable after costs

out-of-sample valid

13. A Histogram Shows How Outcomes Are Spread

A histogram groups observations into return intervals.

return values
↓
divide into bins
↓
count how much of each group
falls inside each bin

Because the baseline normally has many more observations, the program converts each group's histogram into percent of that group.

That makes shape comparison more useful than raw counts.

14. What Should You Look for in the Histogram?

Ask questions such as:

Is one group shifted right?

Is one distribution wider?

Does one group have
a longer negative tail?

Are a few extreme gains
pulling the mean upward?

Do both groups overlap heavily?

Do not force the chart to produce a simple yes/no answer.

15. What Is an ECDF?

ECDF stands for:

Empirical
Cumulative
Distribution
Function

The name sounds technical, but the construction is simple.

Sort the observations:

-2%
 0%
+1%
+3%

Then attach cumulative fractions:

-2% → 25%
 0% → 50%
+1% → 75%
+3% → 100%

16. How to Read an ECDF

Pick a return on the horizontal axis.

The vertical value tells us:

What fraction of observations
were less than or equal
to this return?

For example:

ECDF at 0% = 0.40

means

40% of observations
were <= 0%

Therefore roughly 60% were above zero, ignoring exact-zero conventions for this simple interpretation.

ECDF comparing five-bar forward returns after SMA20 Cross Above signals with all eligible AAPL bars — Figure 2. ECDF comparison of five-bar forward returns after SMA20 Cross Above events and all eligible AAPL bars.

Read this chart from left to right. At any return level on the horizontal axis, the vertical value shows the fraction of observations at or below that return. The two lines let us compare the entire cumulative distribution rather than only the mean.

17. Why Use Both Histogram and ECDF?

Histogram
→ intuitive shape
→ tails and clusters

ECDF
→ cumulative comparison
→ no need to choose visual bin boundaries

They show the same observations from different angles.

18. Build the Descriptive Summary in Python

def summarize_returns(values):
    ...

    return {
        "Count": len(valid_values),
        "Mean": mean(valid_values),
        "Median": median(valid_values),
        "Positive Rate": positive_count / len(valid_values),
        "Minimum": min(valid_values),
        "Maximum": max(valid_values),
    }

One function creates the same measurements for both groups.

19. Build the ECDF from Scratch

def empirical_cdf(values):
    clean_values = sorted(
        float(value)
        for value in values
        if value is not None
    )

    n = len(clean_values)

    probabilities = [
        (i + 1) / n
        for i in range(n)
    ]

    return clean_values, probabilities

No statistical package is needed to understand the construction.

20. Validate the Distribution Code Before AAPL

The self-test uses:

-2%
 0%
+1%
+3%

Expected:

Count
= 4

Mean
= +0.5%

Median
= +0.5%

Positive Rate
= 50%

ECDF
25%, 50%, 75%, 100%

If those values fail, the program stops before market interpretation.

21. What Files Does the Program Create?

signal_vs_baseline_distribution_summary.csv

signal_vs_baseline_difference.csv

signal_vs_baseline_raw_returns.csv

signal_vs_baseline_histogram.png

signal_vs_baseline_ecdf.png

The CSV files let us inspect the numbers directly. The two images show the distributions visually.

22. The Baseline Contains Signal Bars Too

Our baseline is:

All Eligible Bars

That includes the relatively small number of Cross Above bars.

This is intentional.

It is an unconditional market baseline:

What happened after
an ordinary eligible bar?

vs

What happened after
a Cross Above bar?

The two samples therefore should not be imagined as two independent randomized experimental groups.

23. There Is Another Dependence Problem

Five-bar forward-return windows can overlap.

bar 1 outcome
→ bars 2 to 6

bar 2 outcome
→ bars 3 to 7

Most of their future period is shared.

Therefore:

many return rows
does not automatically mean
many independent observations

This becomes the central topic of Phase 4-04.

24. What This Lesson Can and Cannot Tell Us

Phase 4-03 can tell us:

how many observations exist

where their center lies

how often they are positive

how wide the outcomes appear

whether tails and outliers matter

how Signal and Baseline
look descriptively different

It cannot yet tell us:

whether the difference
is statistically reliable

how much uncertainty
surrounds the difference

whether observations
are sufficiently independent

whether the rule survives
out-of-sample testing

25. Change One Thing Yourself

Keep the signal rule frozen.

Change only:

forward_horizon = 5

to:

forward_horizon = 1

or:

forward_horizon = 20

Ask:

Does the center move?

Does the spread widen?

Do the tails change?

Does Signal still look
different from Baseline?

This is an observation exercise, not parameter optimization.

26. The Complete Python Program

This lesson introduces no new external package. It reuses FinanceDataReader, pandas, and matplotlib.

Save as:

phase4_03_signal_vs_baseline_distribution.py

from pathlib import Path
from datetime import date, timedelta
from statistics import mean, median
import os

import FinanceDataReader as fdr
import matplotlib.pyplot as plt
import pandas as pd


# ============================================================
# Phase 4-03 — Signal vs Baseline Distribution
# Compare forward-return distributions with Python
# ============================================================


# ------------------------------------------------------------
# 1. Settings
# ------------------------------------------------------------

symbol = "AAPL"

recent_trading_days = 800

sma_period = 20

forward_horizon = 5

signal_color = "seagreen"
baseline_color = "dimgray"

histogram_bins = 24


# ------------------------------------------------------------
# 2. Working folder
# ------------------------------------------------------------

SCRIPT_DIR = Path(__file__).resolve().parent
os.chdir(SCRIPT_DIR)

print("Working folder:")
print(SCRIPT_DIR)
print()


# ------------------------------------------------------------
# 3. Reuse Phase 4-01 blocks
# ------------------------------------------------------------

def simple_moving_average(
    values,
    period,
):
    result = [None] * len(values)

    if period <= 0:
        raise ValueError(
            "period must be positive"
        )

    for i in range(
        period - 1,
        len(values),
    ):
        window = values[
            i - period + 1 : i + 1
        ]

        result[i] = (
            sum(float(v) for v in window)
            / period
        )

    return result


def above_sma_state(
    close_values,
    sma_values,
):
    result = [None] * len(close_values)

    for i in range(len(close_values)):
        if sma_values[i] is None:
            continue

        result[i] = (
            float(close_values[i])
            > float(sma_values[i])
        )

    return result


def crossover_events(
    above_state,
):
    cross_above = [False] * len(above_state)
    cross_below = [False] * len(above_state)

    for i in range(
        1,
        len(above_state),
    ):
        previous_state = above_state[i - 1]
        current_state = above_state[i]

        if (
            previous_state is None
            or current_state is None
        ):
            continue

        cross_above[i] = (
            current_state is True
            and previous_state is False
        )

        cross_below[i] = (
            current_state is False
            and previous_state is True
        )

    return (
        cross_above,
        cross_below,
    )


# ------------------------------------------------------------
# 4. Reuse Phase 4-02 execution-aligned return
# ------------------------------------------------------------

def next_open_forward_return(
    open_values,
    horizon,
):
    result = [None] * len(open_values)

    if horizon <= 0:
        raise ValueError(
            "horizon must be positive"
        )

    last_signal_index = (
        len(open_values)
        - horizon
        - 2
    )

    for i in range(
        0,
        last_signal_index + 1,
    ):
        entry_index = i + 1

        exit_index = (
            entry_index
            + horizon
        )

        entry_open = float(
            open_values[entry_index]
        )

        exit_open = float(
            open_values[exit_index]
        )

        result[i] = (
            exit_open
            / entry_open
            - 1.0
        )

    return result


# ------------------------------------------------------------
# 5. Descriptive statistics
# ------------------------------------------------------------

def summarize_returns(
    values,
):
    valid_values = [
        float(value)
        for value in values
        if value is not None
    ]

    if len(valid_values) == 0:
        return {
            "Count": 0,
            "Mean": None,
            "Median": None,
            "Positive Rate": None,
            "Minimum": None,
            "Maximum": None,
        }

    positive_count = sum(
        value > 0.0
        for value in valid_values
    )

    return {
        "Count": len(valid_values),
        "Mean": mean(valid_values),
        "Median": median(valid_values),
        "Positive Rate": (
            positive_count
            / len(valid_values)
        ),
        "Minimum": min(valid_values),
        "Maximum": max(valid_values),
    }


# ------------------------------------------------------------
# 6. ECDF
# ------------------------------------------------------------

def empirical_cdf(
    values,
):
    clean_values = sorted(
        float(value)
        for value in values
        if value is not None
    )

    n = len(clean_values)

    if n == 0:
        return [], []

    probabilities = [
        (i + 1) / n
        for i in range(n)
    ]

    return (
        clean_values,
        probabilities,
    )


# ------------------------------------------------------------
# 7. Deterministic self-test
# ------------------------------------------------------------

toy_values = [
    -0.02,
    0.00,
    0.01,
    0.03,
]

toy_summary = summarize_returns(
    toy_values
)

assert toy_summary["Count"] == 4

assert abs(
    toy_summary["Mean"]
    - 0.005
) < 1e-12

assert abs(
    toy_summary["Median"]
    - 0.005
) < 1e-12

assert abs(
    toy_summary["Positive Rate"]
    - 0.50
) < 1e-12

toy_x, toy_y = empirical_cdf(
    toy_values
)

assert toy_x == [
    -0.02,
    0.00,
    0.01,
    0.03,
]

assert toy_y == [
    0.25,
    0.50,
    0.75,
    1.00,
]

print("Self-test")
print("=========")
print()
print("Toy summary:")
print(toy_summary)
print()
print("Toy ECDF x:")
print(toy_x)
print()
print("Toy ECDF y:")
print(toy_y)
print()
print("Self-test: PASS")
print()


# ------------------------------------------------------------
# 8. Download market data
# ------------------------------------------------------------

today = date.today()

start_date = (
    today
    - timedelta(days=1800)
).strftime("%Y-%m-%d")

end_date = today.strftime(
    "%Y-%m-%d"
)

df = fdr.DataReader(
    symbol,
    start_date,
    end_date,
)

df = df.tail(
    recent_trading_days
).copy()

if len(df) < (
    sma_period
    + forward_horizon
    + 10
):
    raise ValueError(
        "Not enough market data."
    )


# ------------------------------------------------------------
# 9. Rebuild the SMA20 Cross Above event
# ------------------------------------------------------------

close_values = [
    float(value)
    for value in df["Close"]
]

open_values = [
    float(value)
    for value in df["Open"]
]

sma_values = simple_moving_average(
    close_values,
    period=sma_period,
)

state = above_sma_state(
    close_values,
    sma_values,
)

cross_above, cross_below = (
    crossover_events(
        state
    )
)

df["SMA"] = sma_values
df["Cross Above"] = cross_above
df["Cross Below"] = cross_below


# ------------------------------------------------------------
# 10. Add one frozen forward-return horizon
# ------------------------------------------------------------

return_column = (
    f"Open Fwd {forward_horizon}"
)

df[return_column] = (
    next_open_forward_return(
        open_values,
        horizon=forward_horizon,
    )
)


# ------------------------------------------------------------
# 11. Build the two comparison groups
# ------------------------------------------------------------

signal_values = []

for value in df.loc[
    df["Cross Above"],
    return_column,
]:
    if pd.notna(value):
        signal_values.append(
            float(value)
        )

baseline_values = []

for value in df[
    return_column
]:
    if pd.notna(value):
        baseline_values.append(
            float(value)
        )


# ------------------------------------------------------------
# 12. Descriptive summaries
# ------------------------------------------------------------

signal_summary = summarize_returns(
    signal_values
)

baseline_summary = summarize_returns(
    baseline_values
)

summary_rows = [
    {
        "Group": "Cross Above",
        **signal_summary,
    },
    {
        "Group": "All Eligible Bars",
        **baseline_summary,
    },
]

summary_df = pd.DataFrame(
    summary_rows
)

summary_file = (
    SCRIPT_DIR
    / "signal_vs_baseline_distribution_summary.csv"
)

summary_df.to_csv(
    summary_file,
    index=False,
)


# ------------------------------------------------------------
# 13. Difference table
# ------------------------------------------------------------

difference_row = {
    "Metric": "Signal - Baseline",
    "Mean Difference": (
        signal_summary["Mean"]
        - baseline_summary["Mean"]
    ),
    "Median Difference": (
        signal_summary["Median"]
        - baseline_summary["Median"]
    ),
    "Positive Rate Difference": (
        signal_summary["Positive Rate"]
        - baseline_summary["Positive Rate"]
    ),
}

difference_df = pd.DataFrame(
    [difference_row]
)

difference_file = (
    SCRIPT_DIR
    / "signal_vs_baseline_difference.csv"
)

difference_df.to_csv(
    difference_file,
    index=False,
)


# ------------------------------------------------------------
# 14. Print the comparison
# ------------------------------------------------------------

print("Signal vs Baseline")
print("==================")
print()

display_summary = summary_df.copy()

for column_name in [
    "Mean",
    "Median",
    "Positive Rate",
    "Minimum",
    "Maximum",
]:
    display_summary[column_name] = (
        100.0
        * display_summary[column_name]
    )

print(
    display_summary.to_string(
        index=False
    )
)

print()
print("Differences")
print("===========")
print()

display_difference = (
    difference_df.copy()
)

for column_name in [
    "Mean Difference",
    "Median Difference",
    "Positive Rate Difference",
]:
    display_difference[column_name] = (
        100.0
        * display_difference[column_name]
    )

print(
    display_difference.to_string(
        index=False
    )
)

print()
print(
    "Important:"
)

print(
    "A visible difference is descriptive evidence, "
    "not yet statistical proof."
)

print(
    "This lesson does not calculate "
    "confidence intervals or p-values."
)

print(
    "The forward-return windows may overlap, "
    "so observations are not automatically independent."
)


# ------------------------------------------------------------
# 15. Histogram
# ------------------------------------------------------------

signal_percent = [
    100.0 * value
    for value in signal_values
]

baseline_percent = [
    100.0 * value
    for value in baseline_values
]

fig, ax = plt.subplots(
    figsize=(10, 6),
)

baseline_weights = [
    100.0 / len(baseline_percent)
    for _ in baseline_percent
]

signal_weights = [
    100.0 / len(signal_percent)
    for _ in signal_percent
]

ax.hist(
    baseline_percent,
    bins=histogram_bins,
    weights=baseline_weights,
    alpha=0.45,
    color=baseline_color,
    label="All Eligible Bars",
)

ax.hist(
    signal_percent,
    bins=histogram_bins,
    weights=signal_weights,
    alpha=0.55,
    color=signal_color,
    label="Cross Above",
)

ax.axvline(
    0.0,
    linewidth=1.0,
)

ax.set_title(
    f"{symbol} — {forward_horizon}-Bar "
    "Forward Return Distribution"
)

ax.set_xlabel(
    "Forward Return (%)"
)

ax.set_ylabel(
    "Percent of Group (%)"
)

ax.grid(
    axis="y",
    alpha=0.20,
)

ax.legend()

fig.subplots_adjust(
    left=0.10,
    right=0.97,
    top=0.90,
    bottom=0.13,
)

histogram_file = (
    SCRIPT_DIR
    / "signal_vs_baseline_histogram.png"
)

fig.savefig(
    histogram_file,
    dpi=140,
)

plt.close(fig)


# ------------------------------------------------------------
# 16. ECDF comparison
# ------------------------------------------------------------

signal_ecdf_x, signal_ecdf_y = (
    empirical_cdf(
        signal_percent
    )
)

baseline_ecdf_x, baseline_ecdf_y = (
    empirical_cdf(
        baseline_percent
    )
)

fig, ax = plt.subplots(
    figsize=(10, 6),
)

ax.step(
    baseline_ecdf_x,
    baseline_ecdf_y,
    where="post",
    color=baseline_color,
    linewidth=1.8,
    label="All Eligible Bars",
)

ax.step(
    signal_ecdf_x,
    signal_ecdf_y,
    where="post",
    color=signal_color,
    linewidth=1.8,
    label="Cross Above",
)

ax.axvline(
    0.0,
    linewidth=1.0,
)

ax.set_title(
    f"{symbol} — {forward_horizon}-Bar "
    "Forward Return ECDF"
)

ax.set_xlabel(
    "Forward Return (%)"
)

ax.set_ylabel(
    "Cumulative Probability"
)

ax.set_ylim(
    0.0,
    1.02,
)

ax.grid(
    axis="both",
    alpha=0.20,
)

ax.legend()

fig.subplots_adjust(
    left=0.10,
    right=0.97,
    top=0.90,
    bottom=0.13,
)

ecdf_file = (
    SCRIPT_DIR
    / "signal_vs_baseline_ecdf.png"
)

fig.savefig(
    ecdf_file,
    dpi=140,
)

plt.close(fig)


# ------------------------------------------------------------
# 17. Save raw comparison data
# ------------------------------------------------------------

raw_rows = []

for value in signal_values:
    raw_rows.append(
        {
            "Group": "Cross Above",
            "Forward Return": value,
        }
    )

for value in baseline_values:
    raw_rows.append(
        {
            "Group": "All Eligible Bars",
            "Forward Return": value,
        }
    )

raw_df = pd.DataFrame(
    raw_rows
)

raw_file = (
    SCRIPT_DIR
    / "signal_vs_baseline_raw_returns.csv"
)

raw_df.to_csv(
    raw_file,
    index=False,
)


# ------------------------------------------------------------
# 18. Finish
# ------------------------------------------------------------

print()
print("Files saved:")
print(summary_file)
print(difference_file)
print(raw_file)
print(histogram_file)
print(ecdf_file)

27. Run the Program

python phase4_03_signal_vs_baseline_distribution.py

First confirm:

Self-test: PASS

Then inspect both the summary tables and the two distribution charts.

Research Checkpoint

Signal
SMA20 Cross Above

Outcome
5-bar next-Open forward return

Baseline
All Eligible Bars

Comparison
Count
Mean
Median
Positive Rate
Minimum / Maximum
Histogram
ECDF

Current Status
DESCRIPTIVE COMPARISON

Not Yet
independence check
confidence interval
bootstrap
backtest
costs
out-of-sample test

Check Your Understanding

A mean is only one feature of a return distribution.
Median, positive rate, sample count, and extremes answer different questions.
The same forward-return formula must be used for the signal and baseline groups.
A histogram helps reveal clusters, spread, and tails.
An ECDF shows the fraction of observations at or below each return level.
A positive Signal-minus-Baseline difference is descriptive evidence, not proof of edge.
The All Eligible Bars baseline is unconditional and contains the signal bars as a small subset.
Forward-return windows may overlap, so the observations are not automatically independent.
Distribution comparison should come before statistical confidence claims.

What You Just Learned

Cross Above Event
       ↓
5-bar Forward Return
       ↓
Signal Group
       ↘
        compare
       ↗
Baseline Group
       ↓
Count
Mean
Median
Positive Rate
Histogram
ECDF
       ↓
describe the difference

not yet:

prove the difference

Before asking whether a signal has an edge, examine the full outcome distribution and understand what is creating the average.

Where Do We Go Next?

We now have many forward-return observations.

But there is a problem:

Are these observations
really independent?

If five-bar or twenty-bar windows overlap, several rows can contain much of the same future market movement.

The next lesson investigates that problem before we estimate statistical uncertainty.

Sources and Further Reading

Matplotlib — hist — histogram construction and weighting.
Matplotlib — step — used to draw the ECDF as a step function.
Python Standard Library — statistics — mean and median used in the descriptive summary.

← Previous: What Is a Forward Return? Measure What Happened After a Signal with Python

↑ Back to Trading Research

Next: Why Do Overlapping Forward Returns Matter? Understand Dependent Observations with Python →

Alphesta

Search This Blog