A chart can look convincing even when the data underneath it has a problem.
A duplicate date, a missing value, or rows in the wrong order can quietly affect an indicator or a backtest later.
So before we calculate anything, we will build a small habit: check the data first.
What you will finish:
you will inspect the size of an AAPL DataFrame, confirm the date order,
count duplicate dates, count missing values, and print one final
Data ready result.
1. What Are We Checking?
We will ask four simple questions:
1. How many rows and columns are there?
2. Are the dates in order?
3. Are any dates duplicated?
4. Are any values missing?We are not fixing problems yet. First, we want to learn how to see them.
2. Create a New Python File
In your alphesta-lab folder, create:
check_market_data.py3. Download a Small AAPL Dataset
Start with familiar code:
import FinanceDataReader as fdr
symbol = "AAPL"
start_date = "2025-01-02"
end_date = "2025-01-31"
df = fdr.DataReader(symbol, start_date, end_date)We use a slightly longer date range than before so that the table contains enough rows to inspect.
4. Check the Size of the DataFrame
Add:
print("Shape:", df.shape)
shape tells you the size of the table.
df.shape
→ (number of rows, number of columns)For example:
(20, 6)would mean:
20 rows
6 columnsYour exact result may differ depending on the available market data and columns.
5. Check the Date Order
For most of our time-series work, we want the Date index to move forward in time.
Add:
date_order_ok = df.index.is_monotonic_increasing
print("Date order OK:", date_order_ok)Read this long name one piece at a time:
df.index
→ the Date index
is_monotonic_increasing
→ are the values staying the same or increasing?
With unique trading dates, a result of True means the rows are ordered
from older dates toward newer dates.
2025-01-02
2025-01-03
2025-01-06
2025-01-07
...
→ True6. Check for Duplicate Dates
A daily market table should normally have only one row for each trading date.
Add:
duplicate_dates = df.index.duplicated().sum()
print("Duplicate dates:", duplicate_dates)Break it down:
df.index.duplicated()
→ mark repeated Date values as True
.sum()
→ count how many True values there areThe simplest result to hope for is:
Duplicate dates: 0That means no repeated date was detected in the index.
7. Check for Missing Values
A missing value is a place where the table does not contain a normal value.
In pandas, you can detect missing values with:
df.isna()That produces True and False values for the cells in the DataFrame. We do not need to print that entire table.
Instead, count the missing values in each column:
missing_values = df.isna().sum()
print("Missing values:")
print(missing_values)Conceptually, the output may look like:
Open 0
High 0
Low 0
Close 0
Volume 0A zero means no missing value was found in that column.
8. Count All Missing Values Together
We can add the column counts together:
total_missing = missing_values.sum()
print("Total missing values:", total_missing)Now we have one number for the whole DataFrame.
Total missing values: 0
→ no missing values detected
Total missing values: 3
→ three cells are missing values9. Create One Final Data-Ready Check
Now combine the three quality checks:
data_ready = (
date_order_ok
and duplicate_dates == 0
and total_missing == 0
)
print("Data ready:", data_ready)Read the logic as:
dates are in order
AND
duplicate date count is zero
AND
missing value count is zero
↓
Data ready: TrueThis does not prove that every price is perfect. It is simply a useful first quality check.
10. Your Complete File
import FinanceDataReader as fdr
symbol = "AAPL"
start_date = "2025-01-02"
end_date = "2025-01-31"
df = fdr.DataReader(symbol, start_date, end_date)
print("Shape:", df.shape)
date_order_ok = df.index.is_monotonic_increasing
print("Date order OK:", date_order_ok)
duplicate_dates = df.index.duplicated().sum()
print("Duplicate dates:", duplicate_dates)
missing_values = df.isna().sum()
print("\nMissing values:")
print(missing_values)
total_missing = missing_values.sum()
print("\nTotal missing values:", total_missing)
data_ready = (
date_order_ok
and duplicate_dates == 0
and total_missing == 0
)
print("Data ready:", data_ready)11. Read the Output as a Checklist
Do not look at the output as random technical information. Read it like a checklist.
Shape: (...)
→ Did I receive a reasonable table?
Date order OK: True
→ Are dates ordered forward?
Duplicate dates: 0
→ Is each date unique?
Total missing values: 0
→ Are the cells complete?
Data ready: True
→ Did these basic checks pass?12. What If Data Ready Is False?
Do not immediately delete or replace data.
A False result means:
stop
↓
identify the problem
↓
decide how to handle it
↓
then continue the analysisDifferent problems need different solutions. Sorting dates, removing duplicate rows, and handling missing values are separate decisions.
We will not hide those decisions inside one automatic cleaning command.
13. Change One Input Yourself
Change:
symbol = "AAPL"to:
symbol = "MSFT"Run the same checks again.
The market data changed. The validation logic did not.
different dataset
+
same checklist
↓
repeatable data checkCheck Your Understanding
df.shapeshows the number of rows and columns.df.index.is_monotonic_increasingchecks whether the index is ordered forward.df.index.duplicated().sum()counts repeated index dates.df.isna().sum()counts missing values by column.- A basic validation check can stop you from analyzing questionable data too early.
What You Just Built
market data
↓
check size
↓
check dates
↓
check duplicates
↓
check missing values
↓
Data ready?This habit matters more as your code becomes more complicated.
An indicator or backtest can only be as trustworthy as the data it receives.
Where Do We Go Next?
Now you know how to detect several common data problems.
The next question is what to do when one of those checks fails. We can learn how to sort rows, remove duplicates, and handle missing values without hiding the decisions from the learner.
← Previous: How to Save and Reload Market Data with CSV in Python
Next: How to Clean Market Data Before Analysis with pandas →