In Phase 4-03, we compared the full distribution of five-bar forward returns after an SMA20 Cross Above event with the distribution from all eligible bars.
Signal Group
vs
Baseline Group
↓
Count
Mean
Median
Positive Rate
Histogram
ECDFThat gave us a better description than a single average.
But one important question is still unresolved:
Do 100 forward-return rows
really represent
100 separate pieces of information?Not necessarily.
Phase 4-04 studies a hidden problem in multi-bar forward returns: overlapping outcome windows.
1. Keep the Previous Research Rule Frozen
We do not invent a new strategy in this lesson.
We keep the same rule from Phase 4-01 through Phase 4-03:
Indicator
SMA 20
Entry Event
previous Close <= previous SMA20
AND
current Close > current SMA20
Signal Time
after Close(t)
Execution-Aligned Outcome
Open(t+1)
→ Open(t+6)
Forward Horizon
5 bars
Baseline
All Eligible BarsOnly one new question changes:
How much do the
future windows overlap?2. What Does a Five-Bar Forward Return Contain?
For a signal at row t, our five-bar execution-aligned outcome is:
Entry
Open(t+1)
Exit
Open(t+6)
Return
Open(t+6)
---------
Open(t+1)
- 1This outcome spans five one-bar return intervals.
t+1 → t+2
t+2 → t+3
t+3 → t+4
t+4 → t+5
t+5 → t+63. Now Move the Starting Row Forward by One Bar
The next eligible row has:
Entry
Open(t+2)
Exit
Open(t+7)Its five intervals are:
t+2 → t+3
t+3 → t+4
t+4 → t+5
t+5 → t+6
t+6 → t+7Compare the two windows.
First window
t+1 → t+2
t+2 → t+3
t+3 → t+4
t+4 → t+5
t+5 → t+6
Next window
t+2 → t+3
t+3 → t+4
t+4 → t+5
t+5 → t+6
t+6 → t+7Four of the five one-bar intervals are shared.
gap = 1 bar
horizon = 5 bars
overlap
= 4 bars
= 80%4. See the Calculation on Candlesticks
Before discussing dependence statistically, it helps to see exactly which candles are used in the calculation.
The updated Python program creates:
overlap_window_candlestick.pngThe chart selects an actual SMA20 Cross Above event, shows green bullish candles and red bearish candles, and marks the signal bar:
Signal
known after Close(t)The first five-bar forward-return window is:
Window A
Entry
Open(t+1)
Exit
Open(t+6)On the same candlestick chart, the next eligible row is also shown:
Window B
Entry
Open(t+2)
Exit
Open(t+7)Now the overlap becomes visible rather than abstract.
Window A
t+1 → t+6
Window B
t+2 → t+7
shared intervals
t+2 → t+3
t+3 → t+4
t+4 → t+5
t+5 → t+6
4 of 5 intervals
= 80% overlapThe same example is also saved numerically as:
overlap_window_example.csvIt contains the actual reference date, entry date, entry Open, exit date, exit Open, and forward return for both windows.
This chart is a visual verification tool. The green/red candle colors make the local price movement easier to read, and the figure helps us confirm exactly what the formulas are measuring before we interpret the overlap statistics.
5. Why Does Overlap Matter?
If two observations contain much of the same future market movement, they should not automatically be treated as two independent experiments.
many rows
≠
many independent observationsThis matters because statistical uncertainty depends not only on the number of rows in a table, but also on how much genuinely separate information those rows contain.
Phase 4-04 does not calculate a confidence interval yet. It first diagnoses the dependence problem before Phase 4-05 tries to measure uncertainty.
6. The Baseline Has the Most Obvious Overlap
Our Phase 4-03 baseline was:
All Eligible BarsThat means the baseline normally includes one forward return for almost every eligible row.
With a five-bar horizon, consecutive baseline rows are one bar apart.
gap = 1
overlap fraction
= (5 - 1) / 5
= 80%So a large baseline sample can contain heavy mechanical overlap.
7. Signal Events May Overlap Less Often
SMA20 Cross Above events are usually much less frequent than ordinary bars.
Suppose consecutive signal events are eight bars apart:
gap = 8
horizon = 5
overlap = 0But if two events are only three bars apart:
gap = 3
horizon = 5
overlap
= 5 - 3
= 2 bars
overlap fraction
= 40%Therefore the signal and baseline groups can have very different dependence structures.
8. Build a Simple Overlap Formula
Overlap Bars
=
max(
0,
horizon - gap
)For a five-bar horizon:
gap 1 → overlap 4 → 80%
gap 2 → overlap 3 → 60%
gap 3 → overlap 2 → 40%
gap 4 → overlap 1 → 20%
gap 5 → overlap 0 → 0%
gap 6 → overlap 0 → 0%9. Implement the Formula in Python
def overlap_bars_from_gap(
gap_bars,
horizon,
):
if gap_bars is None:
return None
return max(
0,
horizon - gap_bars,
)Then convert the shared interval count into a fraction:
def overlap_fraction_from_gap(
gap_bars,
horizon,
):
overlap_bars = (
overlap_bars_from_gap(
gap_bars,
horizon,
)
)
if overlap_bars is None:
return None
return (
overlap_bars
/ horizon
)10. Test the Overlap Logic Before AAPL
Use a five-bar horizon and two rows one bar apart.
gap = 1
horizon = 5
expected overlap bars
= 4
expected overlap fraction
= 0.80The program checks:
assert overlap_bars_from_gap(
gap_bars=1,
horizon=5,
) == 4This is another implementation test: confirm the definition before interpreting market data.
11. Build an Overlap Diagnostic Table
For every candidate row, the program records:
Date
Row Index
Gap From Previous
Overlap Bars
Overlap Fraction
Forward Return
Kept Non-OverlappingTwo CSV files are created:
signal_overlap_diagnostic.csv
baseline_overlap_diagnostic.csvThe first lets us inspect spacing between Cross Above events. The second exposes how strongly the all-bars baseline overlaps.
12. What Does “Non-Overlapping” Mean Here?
We now create a simple diagnostic sample.
Keep the first candidate row. Then keep the next row only if it is at least five bars after the last kept row.
keep first row
then require:
next_index
-
last_kept_index
>= horizonThis removes mechanical overlap in the five one-bar return intervals between selected windows.
13. Build the Greedy Non-Overlapping Selector
def greedy_non_overlapping_indices(
candidate_indices,
horizon,
):
selected = []
last_selected = None
for index_value in candidate_indices:
if (
last_selected is None
or index_value - last_selected
>= horizon
):
selected.append(index_value)
last_selected = index_value
return selected14. Work Through a Tiny Selection Example
Suppose candidate rows are:
0
1
5
7
12With a five-bar horizon:
0 → keep
1 → skip
gap from kept row = 1
5 → keep
gap from kept row = 5
7 → skip
gap from kept row = 2
12 → keep
gap from kept row = 7Result:
[0, 5, 12]15. Non-Overlapping Does Not Mean Independent
This distinction is essential.
no mechanical window overlap
≠
statistical independence provenTwo non-overlapping market windows can still be related because markets themselves can exhibit persistence, volatility clustering, common regimes, and other time dependence.
Our non-overlapping sample is therefore a diagnostic, not a complete statistical correction.
16. Why Not Solve Everything with a Statistical Method Now?
There are econometric methods designed for autocorrelation and overlapping-horizon problems.
But introducing them now would mix several new ideas at once.
Phase 4-04
→ understand the dependence problem
Phase 4-05
→ measure uncertainty carefullyThe learning rule remains:
change one thing at a time17. Compare Full and Non-Overlapping Samples
The script calculates descriptive statistics for four groups:
Signal — All
Signal — Non-Overlapping Diagnostic
Baseline — All
Baseline — Non-Overlapping DiagnosticFor each group:
Count
Mean
Median
Positive Rate18. Expect the Sample Count to Fall
Removing mechanical overlap deliberately throws away rows.
more spacing
↓
fewer observationsThat is not automatically bad.
A large table of highly overlapping outcomes can look more informative than it really is.
19. Ask Whether the Descriptive Difference Changes
Phase 4-03 calculated:
Signal Mean
-
Baseline MeanPhase 4-04 calculates it twice:
All Eligible Sampling
vs
Non-Overlapping Diagnostic SamplingIf the difference changes substantially, that is a warning that the previous descriptive result was sensitive to sampling structure.
If the difference stays similar, that is useful descriptive information, but it still does not prove an edge.
20. The Second Chart Shows Signal Spacing
The program creates:
signal_gap_distribution.pngThe horizontal axis measures bars between consecutive Cross Above events.
A vertical line marks the five-bar horizon.
gap < 5
→ adjacent signal windows overlap
gap >= 5
→ no mechanical return-interval overlap
21. The Third Chart Tests Sampling Sensitivity
The second file is:
overlap_mean_difference_comparison.pngIt compares:
Signal - Baseline Mean Return
using:
All Eligible rows
vs
Greedy Non-Overlapping
Diagnostic rows
22. Be Careful with the Baseline Sample
The all-bars baseline and the signal group do not necessarily have the same event frequency.
Therefore they can also have different overlap patterns.
Signal Group
→ sparse events
Baseline Group
→ nearly every eligible rowThis is another reason a simple row count is not enough to understand the strength of the evidence.
23. What This Lesson Can Tell Us
how far apart signal events occur
how many adjacent windows overlap
what fraction of each horizon overlaps
how many rows remain
after simple spacing
whether descriptive results
change after removing
mechanical overlap24. What This Lesson Still Cannot Tell Us
the correct standard error
a confidence interval
a p-value
true statistical independence
out-of-sample robustness
strategy profitability after costsThose questions belong later in the research sequence.
25. The Complete Python Program
This lesson introduces no new external package.
It reuses FinanceDataReader, pandas,
and matplotlib.
Save as:
phase4_04_overlapping_forward_returns.pyfrom pathlib import Path
from datetime import date, timedelta
from statistics import mean, median
import os
import FinanceDataReader as fdr
import matplotlib.pyplot as plt
from matplotlib.patches import Rectangle
import pandas as pd
# ============================================================
# Phase 4-04 — Overlapping Forward Returns
# Diagnose dependent observations before statistical inference
# ============================================================
# ------------------------------------------------------------
# 1. Settings
# ------------------------------------------------------------
symbol = "AAPL"
recent_trading_days = 800
sma_period = 20
forward_horizon = 5
SCRIPT_DIR = Path(__file__).resolve().parent
os.chdir(SCRIPT_DIR)
# ------------------------------------------------------------
# 2. Reuse the frozen Phase 4 signal
# ------------------------------------------------------------
def simple_moving_average(values, period):
result = [None] * len(values)
if period <= 0:
raise ValueError("period must be positive")
for i in range(period - 1, len(values)):
window = values[i - period + 1 : i + 1]
result[i] = sum(float(v) for v in window) / period
return result
def above_sma_state(close_values, sma_values):
result = [None] * len(close_values)
for i in range(len(close_values)):
if sma_values[i] is None:
continue
result[i] = (
float(close_values[i])
> float(sma_values[i])
)
return result
def crossover_events(above_state):
cross_above = [False] * len(above_state)
cross_below = [False] * len(above_state)
for i in range(1, len(above_state)):
previous_state = above_state[i - 1]
current_state = above_state[i]
if previous_state is None or current_state is None:
continue
cross_above[i] = (
current_state is True
and previous_state is False
)
cross_below[i] = (
current_state is False
and previous_state is True
)
return cross_above, cross_below
# ------------------------------------------------------------
# 3. Reuse the Phase 4-02 execution-aligned forward return
# ------------------------------------------------------------
def next_open_forward_return(open_values, horizon):
"""
Signal is known after Close(t).
Entry:
Open(t+1)
Exit for an h-bar horizon:
Open(t+1+h)
"""
result = [None] * len(open_values)
if horizon <= 0:
raise ValueError("horizon must be positive")
last_signal_index = len(open_values) - horizon - 2
for i in range(0, last_signal_index + 1):
entry_index = i + 1
exit_index = entry_index + horizon
entry_open = float(open_values[entry_index])
exit_open = float(open_values[exit_index])
result[i] = exit_open / entry_open - 1.0
return result
# ------------------------------------------------------------
# 4. Descriptive statistics
# ------------------------------------------------------------
def summarize_returns(values):
valid_values = [
float(value)
for value in values
if value is not None
]
if not valid_values:
return {
"Count": 0,
"Mean": None,
"Median": None,
"Positive Rate": None,
}
positive_count = sum(
value > 0.0
for value in valid_values
)
return {
"Count": len(valid_values),
"Mean": mean(valid_values),
"Median": median(valid_values),
"Positive Rate": positive_count / len(valid_values),
}
# ------------------------------------------------------------
# 5. Measure overlap between adjacent candidate windows
# ------------------------------------------------------------
def overlap_bars_from_gap(gap_bars, horizon):
"""
A horizon-h forward return contains h one-bar return intervals.
If two signal rows are gap_bars apart:
gap = 1, horizon = 5
-> 4 of the 5 return intervals overlap
gap = 5
-> 0 return intervals overlap
"""
if gap_bars is None:
return None
if gap_bars < 0:
raise ValueError("gap_bars cannot be negative")
return max(0, horizon - gap_bars)
def overlap_fraction_from_gap(gap_bars, horizon):
overlap_bars = overlap_bars_from_gap(
gap_bars,
horizon,
)
if overlap_bars is None:
return None
return overlap_bars / horizon
# ------------------------------------------------------------
# 6. Build a simple non-overlapping diagnostic sample
# ------------------------------------------------------------
def greedy_non_overlapping_indices(candidate_indices, horizon):
"""
Keep the first candidate, then keep the next candidate only
when it is at least 'horizon' rows after the last kept one.
This removes mechanical overlap in the h one-bar return
intervals between selected windows.
Important:
This is a diagnostic sample, not proof of statistical
independence.
"""
selected = []
last_selected = None
for index_value in candidate_indices:
index_value = int(index_value)
if (
last_selected is None
or index_value - last_selected >= horizon
):
selected.append(index_value)
last_selected = index_value
return selected
# ------------------------------------------------------------
# 7. Diagnostic table
# ------------------------------------------------------------
def build_overlap_table(
market_df,
candidate_indices,
return_column,
horizon,
):
selected_indices = set(
greedy_non_overlapping_indices(
candidate_indices,
horizon,
)
)
rows = []
previous_index = None
for index_value in candidate_indices:
index_value = int(index_value)
if previous_index is None:
gap_bars = None
else:
gap_bars = index_value - previous_index
overlap_bars = overlap_bars_from_gap(
gap_bars,
horizon,
)
overlap_fraction = overlap_fraction_from_gap(
gap_bars,
horizon,
)
rows.append(
{
"Date": market_df.index[index_value],
"Row Index": index_value,
"Gap From Previous": gap_bars,
"Overlap Bars": overlap_bars,
"Overlap Fraction": overlap_fraction,
"Forward Return": float(
market_df.iloc[index_value][return_column]
),
"Kept Non-Overlapping": (
index_value in selected_indices
),
}
)
previous_index = index_value
return pd.DataFrame(rows)
# ------------------------------------------------------------
# 8. Draw candlesticks for visual verification
# ------------------------------------------------------------
def draw_candlesticks(
ax,
market_df,
body_width=0.62,
):
"""
Simple educational candlestick renderer.
Bullish candle:
seagreen
Bearish candle:
firebrick
No external charting package is required.
"""
bullish_color = "seagreen"
bearish_color = "firebrick"
for x, (_, row) in enumerate(
market_df.iterrows()
):
open_price = float(row["Open"])
high_price = float(row["High"])
low_price = float(row["Low"])
close_price = float(row["Close"])
bullish = (
close_price >= open_price
)
candle_color = (
bullish_color
if bullish
else bearish_color
)
ax.vlines(
x,
low_price,
high_price,
color=candle_color,
linewidth=1.0,
)
body_bottom = min(
open_price,
close_price,
)
body_height = abs(
close_price
- open_price
)
if body_height == 0:
body_height = max(
high_price - low_price,
0.01,
) * 0.02
ax.add_patch(
Rectangle(
(
x - body_width / 2.0,
body_bottom,
),
body_width,
body_height,
facecolor=candle_color,
edgecolor=candle_color,
linewidth=1.0,
alpha=0.90,
)
)
# ------------------------------------------------------------
# 9. Deterministic self-test
# ------------------------------------------------------------
toy_candidate_indices = [0, 1, 5, 7, 12]
toy_selected = greedy_non_overlapping_indices(
toy_candidate_indices,
horizon=5,
)
assert toy_selected == [0, 5, 12]
assert overlap_bars_from_gap(
gap_bars=1,
horizon=5,
) == 4
assert abs(
overlap_fraction_from_gap(
gap_bars=1,
horizon=5,
)
- 0.80
) < 1e-12
assert overlap_bars_from_gap(
gap_bars=5,
horizon=5,
) == 0
toy_open = [
100.0,
101.0,
102.0,
103.0,
104.0,
105.0,
106.0,
107.0,
]
toy_forward = next_open_forward_return(
toy_open,
horizon=2,
)
expected = 104.0 / 102.0 - 1.0
assert abs(
toy_forward[1] - expected
) < 1e-12
print("Self-test")
print("=========")
print("Toy candidates:", toy_candidate_indices)
print("Greedy non-overlapping:", toy_selected)
print("5-bar horizon, 1-bar gap overlap: 80%")
print("Self-test: PASS")
print()
# ------------------------------------------------------------
# 10. Download market data
# ------------------------------------------------------------
today = date.today()
start_date = (
today
- timedelta(days=1800)
).strftime("%Y-%m-%d")
end_date = today.strftime("%Y-%m-%d")
df = fdr.DataReader(
symbol,
start_date,
end_date,
)
df = df.tail(
recent_trading_days
).copy()
if len(df) < sma_period + forward_horizon + 10:
raise ValueError("Not enough market data.")
# ------------------------------------------------------------
# 11. Rebuild the frozen SMA20 Cross Above event
# ------------------------------------------------------------
close_values = [
float(value)
for value in df["Close"]
]
open_values = [
float(value)
for value in df["Open"]
]
sma_values = simple_moving_average(
close_values,
period=sma_period,
)
state = above_sma_state(
close_values,
sma_values,
)
cross_above, cross_below = crossover_events(
state
)
df["SMA"] = sma_values
df["Cross Above"] = cross_above
df["Cross Below"] = cross_below
# ------------------------------------------------------------
# 12. Rebuild the same 5-bar next-Open outcome
# ------------------------------------------------------------
return_column = (
f"Open Fwd {forward_horizon}"
)
df[return_column] = next_open_forward_return(
open_values,
horizon=forward_horizon,
)
# ------------------------------------------------------------
# 13. Candidate rows
# ------------------------------------------------------------
signal_indices = [
i
for i, value in enumerate(df["Cross Above"])
if bool(value)
and pd.notna(df.iloc[i][return_column])
]
baseline_indices = [
i
for i, value in enumerate(df[return_column])
if pd.notna(value)
]
# ------------------------------------------------------------
# 14. Candlestick example — see exactly what is measured
# ------------------------------------------------------------
example_signal_candidates = [
i
for i in signal_indices
if (
i + forward_horizon + 2
< len(df)
)
]
candlestick_chart_file = (
SCRIPT_DIR
/ "overlap_window_candlestick.png"
)
candlestick_example_file = (
SCRIPT_DIR
/ "overlap_window_example.csv"
)
if example_signal_candidates:
# Use the most recent qualifying signal so the example
# is visually close to the end of the downloaded dataset.
example_signal_index = (
example_signal_candidates[-1]
)
# Window A:
# signal at Close(t)
# entry = Open(t+1)
# exit = Open(t+1+h)
window_a_entry_index = (
example_signal_index + 1
)
window_a_exit_index = (
window_a_entry_index
+ forward_horizon
)
# Window B:
# the very next eligible row
# entry = Open(t+2)
# exit = Open(t+2+h)
window_b_reference_index = (
example_signal_index + 1
)
window_b_entry_index = (
window_b_reference_index + 1
)
window_b_exit_index = (
window_b_entry_index
+ forward_horizon
)
window_a_return = (
float(
df.iloc[
window_a_exit_index
]["Open"]
)
/
float(
df.iloc[
window_a_entry_index
]["Open"]
)
- 1.0
)
window_b_return = (
float(
df.iloc[
window_b_exit_index
]["Open"]
)
/
float(
df.iloc[
window_b_entry_index
]["Open"]
)
- 1.0
)
example_rows = [
{
"Window": "A — Signal Row",
"Reference Date": (
df.index[
example_signal_index
]
),
"Entry Date": (
df.index[
window_a_entry_index
]
),
"Entry Open": (
float(
df.iloc[
window_a_entry_index
]["Open"]
)
),
"Exit Date": (
df.index[
window_a_exit_index
]
),
"Exit Open": (
float(
df.iloc[
window_a_exit_index
]["Open"]
)
),
"Forward Return": (
window_a_return
),
},
{
"Window": "B — Next Eligible Row",
"Reference Date": (
df.index[
window_b_reference_index
]
),
"Entry Date": (
df.index[
window_b_entry_index
]
),
"Entry Open": (
float(
df.iloc[
window_b_entry_index
]["Open"]
)
),
"Exit Date": (
df.index[
window_b_exit_index
]
),
"Exit Open": (
float(
df.iloc[
window_b_exit_index
]["Open"]
)
),
"Forward Return": (
window_b_return
),
},
]
pd.DataFrame(
example_rows
).to_csv(
candlestick_example_file,
index=False,
)
plot_start = max(
0,
example_signal_index - 10,
)
plot_end = min(
len(df),
window_b_exit_index + 8,
)
plot_df = df.iloc[
plot_start:plot_end
].copy()
signal_x = (
example_signal_index
- plot_start
)
a_entry_x = (
window_a_entry_index
- plot_start
)
a_exit_x = (
window_a_exit_index
- plot_start
)
b_entry_x = (
window_b_entry_index
- plot_start
)
b_exit_x = (
window_b_exit_index
- plot_start
)
fig, ax = plt.subplots(
figsize=(12, 7)
)
draw_candlesticks(
ax,
plot_df,
)
ax.plot(
list(range(len(plot_df))),
plot_df["SMA"],
linewidth=1.4,
label=f"SMA {sma_period}",
)
ax.axvline(
signal_x,
linewidth=1.5,
label="Signal known after Close(t)",
)
# Window A:
# Open(t+1) -> Open(t+6) for h = 5
ax.axvspan(
a_entry_x,
a_exit_x,
alpha=0.12,
hatch="//",
label=(
"Window A: "
"Open(t+1) → Open(t+6)"
),
)
# Window B:
# Open(t+2) -> Open(t+7)
ax.axvspan(
b_entry_x,
b_exit_x,
alpha=0.08,
hatch="\\\\",
label=(
"Window B: "
"Open(t+2) → Open(t+7)"
),
)
ax.axvline(
a_entry_x,
linestyle="--",
linewidth=1.1,
)
ax.axvline(
a_exit_x,
linestyle="--",
linewidth=1.1,
)
ax.axvline(
b_entry_x,
linestyle=":",
linewidth=1.1,
)
ax.axvline(
b_exit_x,
linestyle=":",
linewidth=1.1,
)
ax.set_title(
f"{symbol} — Where the 5-Bar "
"Forward Returns Are Measured"
)
ax.set_ylabel(
"Price"
)
ax.grid(
axis="y",
alpha=0.20,
)
step = max(
1,
len(plot_df) // 8,
)
tick_positions = list(
range(
0,
len(plot_df),
step,
)
)
tick_labels = [
plot_df.index[i].strftime(
"%Y-%m-%d"
)
for i in tick_positions
]
ax.set_xticks(
tick_positions
)
ax.set_xticklabels(
tick_labels,
rotation=35,
ha="right",
)
ax.legend()
fig.subplots_adjust(
left=0.09,
right=0.98,
top=0.90,
bottom=0.18,
)
fig.savefig(
candlestick_chart_file,
dpi=140,
)
plt.close(fig)
print()
print("Candlestick overlap example")
print("===========================")
print(
"Signal date:",
df.index[
example_signal_index
].strftime("%Y-%m-%d"),
)
print(
"Window A:",
df.index[
window_a_entry_index
].strftime("%Y-%m-%d"),
"Open ->",
df.index[
window_a_exit_index
].strftime("%Y-%m-%d"),
"Open",
)
print(
"Window A return:",
f"{100.0 * window_a_return:.4f}%",
)
print(
"Window B:",
df.index[
window_b_entry_index
].strftime("%Y-%m-%d"),
"Open ->",
df.index[
window_b_exit_index
].strftime("%Y-%m-%d"),
"Open",
)
print(
"Window B return:",
f"{100.0 * window_b_return:.4f}%",
)
print(
"Shared one-bar intervals:",
forward_horizon - 1,
"of",
forward_horizon,
)
print(
"Mechanical overlap:",
f"{100.0 * (forward_horizon - 1) / forward_horizon:.1f}%",
)
# ------------------------------------------------------------
# 15. Build overlap diagnostics
# ------------------------------------------------------------
signal_overlap_df = build_overlap_table(
df,
signal_indices,
return_column,
forward_horizon,
)
baseline_overlap_df = build_overlap_table(
df,
baseline_indices,
return_column,
forward_horizon,
)
signal_overlap_file = (
SCRIPT_DIR
/ "signal_overlap_diagnostic.csv"
)
baseline_overlap_file = (
SCRIPT_DIR
/ "baseline_overlap_diagnostic.csv"
)
signal_overlap_df.to_csv(
signal_overlap_file,
index=False,
)
baseline_overlap_df.to_csv(
baseline_overlap_file,
index=False,
)
# ------------------------------------------------------------
# 16. Count adjacent overlapping windows
# ------------------------------------------------------------
def adjacent_overlap_summary(overlap_df):
comparable = overlap_df[
overlap_df["Gap From Previous"].notna()
].copy()
if len(comparable) == 0:
return {
"Adjacent Pairs": 0,
"Overlapping Pairs": 0,
"Overlap Pair Rate": None,
"Mean Overlap Fraction": None,
}
overlapping_mask = (
comparable["Overlap Bars"] > 0
)
return {
"Adjacent Pairs": len(comparable),
"Overlapping Pairs": int(
overlapping_mask.sum()
),
"Overlap Pair Rate": float(
overlapping_mask.mean()
),
"Mean Overlap Fraction": float(
comparable["Overlap Fraction"].mean()
),
}
signal_overlap_summary = adjacent_overlap_summary(
signal_overlap_df
)
baseline_overlap_summary = adjacent_overlap_summary(
baseline_overlap_df
)
# ------------------------------------------------------------
# 17. Full vs non-overlapping descriptive samples
# ------------------------------------------------------------
signal_all_values = [
float(df.iloc[i][return_column])
for i in signal_indices
]
baseline_all_values = [
float(df.iloc[i][return_column])
for i in baseline_indices
]
signal_nonoverlap_indices = (
greedy_non_overlapping_indices(
signal_indices,
forward_horizon,
)
)
baseline_nonoverlap_indices = (
greedy_non_overlapping_indices(
baseline_indices,
forward_horizon,
)
)
signal_nonoverlap_values = [
float(df.iloc[i][return_column])
for i in signal_nonoverlap_indices
]
baseline_nonoverlap_values = [
float(df.iloc[i][return_column])
for i in baseline_nonoverlap_indices
]
comparison_rows = []
for group_name, values in [
("Signal — All", signal_all_values),
(
"Signal — Non-Overlapping Diagnostic",
signal_nonoverlap_values,
),
("Baseline — All", baseline_all_values),
(
"Baseline — Non-Overlapping Diagnostic",
baseline_nonoverlap_values,
),
]:
row = {
"Group": group_name,
**summarize_returns(values),
}
comparison_rows.append(row)
comparison_df = pd.DataFrame(
comparison_rows
)
comparison_file = (
SCRIPT_DIR
/ "overlap_sampling_comparison.csv"
)
comparison_df.to_csv(
comparison_file,
index=False,
)
# ------------------------------------------------------------
# 18. Signal-minus-baseline mean differences
# ------------------------------------------------------------
signal_all_summary = summarize_returns(
signal_all_values
)
baseline_all_summary = summarize_returns(
baseline_all_values
)
signal_nonoverlap_summary = summarize_returns(
signal_nonoverlap_values
)
baseline_nonoverlap_summary = summarize_returns(
baseline_nonoverlap_values
)
difference_rows = [
{
"Sampling": "All Eligible",
"Signal Mean": signal_all_summary["Mean"],
"Baseline Mean": baseline_all_summary["Mean"],
"Signal - Baseline": (
signal_all_summary["Mean"]
- baseline_all_summary["Mean"]
),
},
{
"Sampling": "Greedy Non-Overlapping Diagnostic",
"Signal Mean": signal_nonoverlap_summary["Mean"],
"Baseline Mean": baseline_nonoverlap_summary["Mean"],
"Signal - Baseline": (
signal_nonoverlap_summary["Mean"]
- baseline_nonoverlap_summary["Mean"]
),
},
]
difference_df = pd.DataFrame(
difference_rows
)
difference_file = (
SCRIPT_DIR
/ "overlap_mean_difference.csv"
)
difference_df.to_csv(
difference_file,
index=False,
)
# ------------------------------------------------------------
# 19. Print overlap diagnostics
# ------------------------------------------------------------
print("Overlap diagnostics")
print("===================")
print()
print(f"Symbol: {symbol}")
print(f"Forward horizon: {forward_horizon} bars")
print()
print("Signal group:")
print(signal_overlap_summary)
print()
print("Baseline group:")
print(baseline_overlap_summary)
print()
print("Full vs non-overlapping descriptive samples")
print("-------------------------------------------")
display_comparison = comparison_df.copy()
for column_name in [
"Mean",
"Median",
"Positive Rate",
]:
display_comparison[column_name] = (
100.0
* display_comparison[column_name]
)
print(
display_comparison.to_string(
index=False
)
)
print()
print("Signal-minus-baseline mean difference")
print("-------------------------------------")
display_difference = difference_df.copy()
for column_name in [
"Signal Mean",
"Baseline Mean",
"Signal - Baseline",
]:
display_difference[column_name] = (
100.0
* display_difference[column_name]
)
print(
display_difference.to_string(
index=False
)
)
print()
print("Important:")
print(
"Removing mechanical overlap is a diagnostic step. "
"It does not prove statistical independence."
)
# ------------------------------------------------------------
# 20. Chart 2 — gap distribution between signal events
# ------------------------------------------------------------
signal_gaps = [
int(value)
for value in signal_overlap_df[
"Gap From Previous"
].dropna()
]
gap_chart_file = (
SCRIPT_DIR
/ "signal_gap_distribution.png"
)
if signal_gaps:
fig, ax = plt.subplots(
figsize=(10, 6)
)
ax.hist(
signal_gaps,
bins=min(20, max(5, len(signal_gaps))),
)
ax.axvline(
forward_horizon,
linestyle="--",
linewidth=1.4,
label=f"{forward_horizon}-bar horizon",
)
ax.set_title(
f"{symbol} — Gaps Between SMA20 Cross Above Events"
)
ax.set_xlabel(
"Bars Between Consecutive Signal Events"
)
ax.set_ylabel(
"Number of Adjacent Signal Pairs"
)
ax.grid(
axis="y",
alpha=0.20,
)
ax.legend()
fig.subplots_adjust(
left=0.11,
right=0.97,
top=0.90,
bottom=0.14,
)
fig.savefig(
gap_chart_file,
dpi=140,
)
plt.close(fig)
# ------------------------------------------------------------
# 21. Chart 3 — full vs non-overlapping mean difference
# ------------------------------------------------------------
difference_chart_file = (
SCRIPT_DIR
/ "overlap_mean_difference_comparison.png"
)
labels = list(
difference_df["Sampling"]
)
values = [
100.0 * float(value)
for value in difference_df[
"Signal - Baseline"
]
]
fig, ax = plt.subplots(
figsize=(10, 6)
)
ax.bar(
list(range(len(labels))),
values,
)
ax.axhline(
0.0,
linewidth=1.0,
)
ax.set_xticks(
list(range(len(labels)))
)
ax.set_xticklabels(
labels,
rotation=12,
ha="right",
)
ax.set_ylabel(
"Signal - Baseline Mean Return (percentage points)"
)
ax.set_title(
f"{symbol} — Does the Descriptive Difference Change "
"After Removing Mechanical Overlap?"
)
ax.grid(
axis="y",
alpha=0.20,
)
fig.subplots_adjust(
left=0.13,
right=0.97,
top=0.88,
bottom=0.22,
)
fig.savefig(
difference_chart_file,
dpi=140,
)
plt.close(fig)
# ------------------------------------------------------------
# 22. Finish
# ------------------------------------------------------------
print()
print("Files saved:")
print(signal_overlap_file)
print(baseline_overlap_file)
print(comparison_file)
print(difference_file)
if example_signal_candidates:
print(candlestick_example_file)
print(candlestick_chart_file)
if signal_gaps:
print(gap_chart_file)
print(difference_chart_file)
26. Run the Program
python phase4_04_overlapping_forward_returns.pyFirst confirm:
Self-test: PASSThen inspect:
signal_overlap_diagnostic.csv
baseline_overlap_diagnostic.csv
overlap_sampling_comparison.csv
overlap_mean_difference.csv
overlap_window_example.csv
overlap_window_candlestick.png
signal_gap_distribution.png
overlap_mean_difference_comparison.png27. Research Checkpoint
Signal
SMA20 Cross Above
Outcome
5-bar next-Open forward return
Baseline
All Eligible Bars
New Question
Do outcome windows overlap?
Diagnostic
Gap between candidate rows
Overlap bars
Overlap fraction
Comparison
All observations
vs
Greedy non-overlapping sample
Current Status
DEPENDENCE DIAGNOSIS
Not Yet
confidence interval
bootstrap
formal inference
backtest
costs
out-of-sample test28. Check Your Understanding
- The candlestick verification chart shows exactly where the signal, entry Open, and exit Open occur.
- A multi-bar forward return covers several future one-bar return intervals.
- Consecutive five-bar forward returns share four of those five intervals.
- Many rows do not automatically represent many independent observations.
- The all-bars baseline can contain much more mechanical overlap than a sparse signal group.
- Overlap can be measured from the gap between candidate rows and the forward horizon.
- A simple spacing rule can create a non-overlapping diagnostic sample.
- Non-overlapping windows do not prove statistical independence.
- Removing overlap can sharply reduce sample count.
- A descriptive signal-minus-baseline difference should be checked for sensitivity to sampling structure.
- Dependence should be understood before making confidence claims.
29. Change One Thing Yourself
Keep the SMA20 Cross Above signal frozen.
Change only:
forward_horizon = 5to:
forward_horizon = 20Then ask:
Does overlap become more common?
How much does the
non-overlapping sample shrink?
Does the Signal - Baseline
descriptive difference change?Do not optimize the horizon. The purpose is to understand how horizon length changes dependence.
30. What You Just Learned
Forward Return
↓
future window
↓
adjacent observations
may share future bars
↓
mechanical overlap
↓
dependent information
↓
diagnose gaps
and overlap fractions
↓
build a simple
non-overlapping sample
↓
compare descriptive results
↓
prepare for
uncertainty estimationBefore treating a large table of forward returns as strong evidence, check how much of the future market path those observations share.
31. Where Do We Go Next?
We now know that the number of rows can exaggerate how much separate information we have.
The next research question is:
How uncertain is
the observed
Signal - Baseline difference?Phase 4-05 will move from descriptive differences toward confidence intervals and bootstrap reasoning.
Sources and Further Reading
- Hansen and Hodrick — Forward Exchange Rates as Optimal Predictors of Future Spot Rates — a classic econometric reference related to inference when time-series errors are serially dependent.
- statsmodels — HAC robust covariance — an example of later tools designed for covariance estimation robust to autocorrelation; not used in this lesson.
- pandas — DataFrame.shift — useful background for aligning present rows with lagged or future observations.