In the previous lesson, you learned how to detect common data problems.
Now we will take the next step: clean the data only when a specific problem needs to be fixed.
This is also where you may notice something important about Phase 1. You have been learning pandas naturally while working with real market data.
DataFrame
→ select columns
→ filter rows
→ save CSV
→ reload CSV
→ check data
→ clean dataThat is intentional. Instead of stopping for a separate pandas course, we are learning each pandas tool when a real task needs it.
What you will finish: you will sort a market DataFrame by date, remove repeated dates, remove rows that are missing key OHLCV values, and run the quality checks again.
1. Start with Real AAPL Data
Create a new file:
clean_market_data.pyStart with:
import FinanceDataReader as fdr
symbol = "AAPL"
start_date = "2025-01-02"
end_date = "2025-01-31"
df = fdr.DataReader(symbol, start_date, end_date)
We will keep the original DataFrame in df
and build a cleaned version called clean_df.
2. Sort the Rows by Date
Add:
clean_df = df.sort_index()
sort_index() sorts rows using the DataFrame index.
In our market data, the index is Date.
df.sort_index()
→ sort by Date index
→ oldest date first
→ newest date last
pandas returns a new sorted DataFrame,
so we store the result in clean_df.
If the data was already sorted correctly, the visible order may not change. That is fine.
3. Find Duplicate Dates
In Phase 1-7, you counted duplicate dates with:
clean_df.index.duplicated().sum()This time, keep the True/False result itself:
duplicate_mask = clean_df.index.duplicated(keep="first")
keep="first" means:
first occurrence
→ keep it
later occurrence of the same date
→ mark it as duplicate4. Remove the Duplicate Dates
Add:
clean_df = clean_df[~duplicate_mask]You already met row filtering in Phase 1-5. This is the same idea.
The new symbol is ~.
Here it means NOT.
duplicate_mask
→ duplicate rows are True
~duplicate_mask
→ duplicate rows become False
clean_df[~duplicate_mask]
→ keep rows that are NOT duplicatesIf the original dataset contained no duplicate dates, the number of rows will simply stay the same.
5. Decide Which Columns Are Essential
Missing values need more thought than sorting or duplicates.
We should not automatically invent a price just because one value is missing. For this beginner workflow, we will treat the five OHLCV columns as required.
required_columns = [
"Open",
"High",
"Low",
"Close",
"Volume"
]These are the columns we want to be complete before building indicators.
6. Check Missing OHLCV Values Before Cleaning
Add:
print("Missing OHLCV values before cleaning:")
print(clean_df[required_columns].isna().sum())Notice how several pandas ideas now connect:
clean_df[required_columns]
→ select several columns
.isna()
→ find missing cells
.sum()
→ count them7. Remove Rows with Missing OHLCV Values
Add:
clean_df = clean_df.dropna(subset=required_columns)
dropna() removes rows containing missing values.
The subset option tells pandas which columns matter for this decision.
dropna(
subset=required_columns
)
→ look only at Open, High, Low, Close, Volume
→ remove a row if one of those required values is missingThis is a deliberate choice.
You may see examples elsewhere that use ffill() or fillna()
to fill missing values.
Those methods are useful in many datasets, but automatically filling OHLCV prices
can create values that were not actually observed in the market.
For now, we will not manufacture missing prices.
8. Compare the Size Before and After
Add:
print("Original shape:", df.shape)
print("Clean shape:", clean_df.shape)If the dataset was already clean, both shapes may be identical.
That does not mean the cleaning code failed. It means there was nothing to remove.
9. Run the Quality Checks Again
Now verify the cleaned result:
date_order_ok = clean_df.index.is_monotonic_increasing
duplicate_dates = clean_df.index.duplicated().sum()
missing_ohlcv = clean_df[required_columns].isna().sum().sum()
print("Date order OK:", date_order_ok)
print("Duplicate dates:", duplicate_dates)
print("Missing OHLCV values:", missing_ohlcv)Then combine the checks:
data_ready = (
date_order_ok
and duplicate_dates == 0
and missing_ohlcv == 0
)
print("Data ready:", data_ready)10. Your Complete File
import FinanceDataReader as fdr
symbol = "AAPL"
start_date = "2025-01-02"
end_date = "2025-01-31"
df = fdr.DataReader(symbol, start_date, end_date)
required_columns = [
"Open",
"High",
"Low",
"Close",
"Volume"
]
clean_df = df.sort_index()
duplicate_mask = clean_df.index.duplicated(keep="first")
clean_df = clean_df[~duplicate_mask]
print("Missing OHLCV values before cleaning:")
print(clean_df[required_columns].isna().sum())
clean_df = clean_df.dropna(subset=required_columns)
print("\nOriginal shape:", df.shape)
print("Clean shape:", clean_df.shape)
date_order_ok = clean_df.index.is_monotonic_increasing
duplicate_dates = clean_df.index.duplicated().sum()
missing_ohlcv = clean_df[required_columns].isna().sum().sum()
print("\nDate order OK:", date_order_ok)
print("Duplicate dates:", duplicate_dates)
print("Missing OHLCV values:", missing_ohlcv)
data_ready = (
date_order_ok
and duplicate_dates == 0
and missing_ohlcv == 0
)
print("Data ready:", data_ready)11. What pandas Skills Did You Use?
This lesson may look like a market-data lesson, but look at the pandas skills underneath it:
sort_index()
→ sort rows
index.duplicated()
→ identify repeated index values
DataFrame[mask]
→ filter rows
DataFrame[columns]
→ select several columns
isna()
→ detect missing values
dropna()
→ remove rows with missing values
shape
→ inspect table sizeThis is why learning pandas inside real market-data tasks works well. Each method has a reason to exist.
12. Change One Input Yourself
Change:
symbol = "AAPL"to:
symbol = "NVDA"Run the same cleaning and validation process.
different stock
+
same preparation process
↓
repeatable research workflowCheck Your Understanding
sort_index()sorts rows by the Date index.index.duplicated()identifies repeated dates.~reverses a True/False mask.dropna(subset=...)removes rows missing required values.- Automatically filling missing OHLCV prices is a decision, not a default rule.
- You should validate the data again after cleaning it.
What You Just Built
raw market data
↓
sort dates
↓
remove duplicate dates
↓
remove incomplete OHLCV rows
↓
check again
↓
analysis-ready DataFramePhase 1 Is Complete
You can now handle the basic data path from the market to an analysis-ready DataFrame.
get data
→ understand the table
→ inspect market listings
→ find a stock
→ save and reload data
→ validate data
→ clean dataAlong the way, you also learned a practical foundation of pandas without treating pandas as a separate subject.
Where Do We Go Next?
Phase 2 begins with indicators.
We will not start by installing a large technical-analysis library. We will begin with a simple indicator that you can calculate yourself and understand line by line.
That is where market data starts becoming technical analysis.