In Pandas – we learnt to **create and access** data. Now in Pandas – II, we go deeper — **analysing, cleaning, and summarising** real data! Think of it as going from knowing how to read a report card, to actually calculating toppers, averages, and finding who was absent! 📊
> [!TIP]
> **How to use these notes:** Run every code example yourself in Python! Focus especially on **Section 2.4 (Descriptive Statistics)**, **Section 2.6 (Missing Data)**, and **Section 2.7 (groupby)** — these appear in EVERY board exam. Simple analogies are used throughout — no stress! 😊
---
## 2.1 🌟 Introduction
In this chapter, we move from just storing data to actually **working with it**:
```mermaid
graph LR
P1["📘 Pandas I\nCreate & Store Data\nSeries & DataFrames"]
P2["📗 Pandas II\nAnalyse & Clean Data\nStats, Iteration, Groups"]
P1 -->|"Next Level"| P2
style P1 fill:#2196F3,color:#fff
style P2 fill:#4CAF50,color:#fff
```
::: grid
::: card 🔄 | Iterating | Visit each row or column one by one | Like reading a register line by line
::: card ➕ | Binary Ops | Math between two DataFrames | Add marks of two tests together
::: card 📊 | Statistics | Mean, Median, Max, Min... | Instantly find class topper or average
::: card 🧹 | Missing Data | Handle NaN (absent students!) | Fill or remove missing entries
::: card 📦 | groupby() | Group data by category | Section-wise or city-wise analysis
:::
**Our Master DataFrame — used throughout this chapter:**
```python
# master_setup.py
import pandas as pd
import numpy as np
data = {
'Name': ['Arjun', 'Bina', 'Chirag', 'Divya', 'Esha'],
'Maths': [90, 85, 78, 92, 88],
'Science': [88, 95, np.nan, 90, 82], # Chirag was absent!
'English': [75, 80, 72, 85, 79],
'Section': ['A', 'A', 'B', 'B', 'A']
}
df = pd.DataFrame(data)
print(df)
```
**Output:**
```
Name Maths Science English Section
0 Arjun 90 88.0 75 A
1 Bina 85 95.0 80 A
2 Chirag 78 NaN 72 B
3 Divya 92 90.0 85 B
4 Esha 88 82.0 79 A
```
> NaN (Not a Number) in Chirag's Science row — he was **absent** for that exam!
---
## 2.2 🔄 Iterating Over a DataFrame
**Iteration** means visiting each part of the DataFrame one by one — like reading a register row by row or column by column.
```mermaid
graph LR
IT["Iterating\nover DataFrame"]
IR["iterrows()\nRow by Row\n(Horizontal)"]
IC["iteritems()\n/ items()\nColumn by Column\n(Vertical)"]
IT --> IR
IT --> IC
style IR fill:#FF9800,color:#fff
style IC fill:#9C27B0,color:#fff
```
---
### 2.2.1 Using `iterrows()` — Row by Row 🔄
`iterrows()` goes through the DataFrame **one row at a time**. Each time, it gives you:
- The **row index** (number or label)
- A **Series** containing all values in that row
> **Analogy:** Imagine a teacher reading the class register — going student by student from top to bottom. That's `iterrows()`!
```python
# iterrows_demo.py
import pandas as pd
import numpy as np
data = {
'Name': ['Arjun', 'Bina', 'Chirag', 'Divya'],
'Maths': [90, 85, 78, 92],
'English': [75, 80, 72, 85]
}
df = pd.DataFrame(data)
print("Reading students one by one:")
for index, row in df.iterrows():
print(f"Row {index}: {row['Name']} → Maths={row['Maths']}, English={row['English']}")
```
**Output:**
```
Reading students one by one:
Row 0: Arjun → Maths=90, English=75
Row 1: Bina → Maths=85, English=80
Row 2: Chirag → Maths=78, English=72
Row 3: Divya → Maths=92, English=85
```
**Practical Example — Find students who passed (Maths ≥ 80):**
```python
# find_passers.py
print("Students who scored ≥ 80 in Maths:")
for index, row in df.iterrows():
if row['Maths'] >= 80:
print(f" ✅ {row['Name']} — {row['Maths']}")
```
**Output:**
```
Students who scored ≥ 80 in Maths:
✅ Arjun — 90
✅ Bina — 85
✅ Divya — 92
```
> [!NOTE]
> **iterrows() is slow for large data! 🧠**
> For small DataFrames (like in CBSE exams), iterrows() is fine. But for millions of rows, it's slow. Real data scientists prefer vectorized operations (like `df[df['Maths'] >= 80]`). For your board exam, always know iterrows() as the correct answer!
---
### 2.2.2 Using `iteritems()` — Column by Column 📊
`iteritems()` goes through the DataFrame **one column at a time**. Each time, it gives you:
- The **column name** (header)
- A **Series** containing all values in that column
> **Analogy:** Instead of reading row by row (student by student), now you're reading column by column (subject by subject) — first all Maths marks, then all English marks...
```python
# iteritems_demo.py
import pandas as pd
data = {
'Maths': [90, 85, 78, 92],
'Science': [88, 95, 70, 90],
'English': [75, 80, 72, 85]
}
df = pd.DataFrame(data)
print("Reading subjects one by one:")
for col_name, col_data in df.items(): # Note: use .items() in newer Pandas
print(f"\nSubject: {col_name}")
print(f" All marks: {col_data.values}")
print(f" Class Average: {col_data.mean():.2f}")
```
**Output:**
```
Reading subjects one by one:
Subject: Maths
All marks: [90 85 78 92]
Class Average: 86.25
Subject: Science
All marks: [88 95 70 90]
Class Average: 85.75
Subject: English
All marks: [75 80 72 85]
Class Average: 78.00
```
> [!WARNING]
> **iteritems() vs items() — Important!**
> `iteritems()` was the old name in Pandas. In **newer versions of Pandas (1.5+)**, it is **deprecated (removed)** and replaced with `items()`. Both do exactly the same thing.
> In CBSE exams: both `iteritems()` and `items()` are accepted in answers!
**Comparison Table:**
| Feature | `iterrows()` | `iteritems()` / `items()` |
| :--- | :--- | :--- |
| **Traversal** | Row by row | Column by column |
| **Direction** | Horizontal (left-right) | Vertical (top-bottom) |
| **Gives you** | (index, Series of row) | (col_name, Series of column) |
| **Use when** | Processing student-wise | Processing subject-wise |
> [!IMPORTANT]
> **Board Exam Tip**
> "What is the difference between iterrows() and iteritems()?" — **2-mark** question!
> Answer: `iterrows()` iterates **row by row** (returns index and row as Series). `iteritems()` iterates **column by column** (returns column name and column as Series).
---
## 2.3 ➕ Binary Operations in a DataFrame
**Binary operations** are mathematical operations (+, -, *, /) performed between **two DataFrames** (or a DataFrame and a number/Series).
> **Analogy:** Imagine you have **Test 1 marks** in one sheet and **Test 2 marks** in another sheet. Adding them gives you **Total marks**. That's a binary operation between two DataFrames!
```mermaid
graph LR
D1["📊 DataFrame 1\nTest 1 Marks"]
D2["📊 DataFrame 2\nTest 2 Marks"]
OP["➕ Binary Operation\n(Addition)"]
RES["📊 Result\nTotal Marks"]
D1 --> OP
D2 --> OP
OP --> RES
style OP fill:#FF9800,color:#fff
style RES fill:#4CAF50,color:#fff
```
**Basic Binary Operations:**
```python
# binary_ops_demo.py
import pandas as pd
test1 = pd.DataFrame({
'Math': [45, 38, 42],
'Science': [40, 45, 35]
}, index=['Arjun', 'Bina', 'Chirag'])
test2 = pd.DataFrame({
'Math': [48, 40, 38],
'Science': [42, 48, 40]
}, index=['Arjun', 'Bina', 'Chirag'])
print("Test 1:\n", test1)
print("\nTest 2:\n", test2)
print("\nTotal (test1 + test2):")
print(test1 + test2)
print("\nDifference (test2 - test1):")
print(test2 - test1)
```
**Output:**
```
Test 1:
Math Science
Arjun 45 40
Bina 38 45
Chirag 42 35
Test 2:
Math Science
Arjun 48 42
Bina 40 48
Chirag 38 40
Total (test1 + test2):
Math Science
Arjun 93 82
Bina 78 93
Chirag 80 75
Difference (test2 - test1):
Math Science
Arjun 3 2
Bina 2 3
Chirag -4 5
```
**What happens when indexes DON'T match? — NaN appears! ⚠️**
```python
# mismatched_binary.py
import pandas as pd
df1 = pd.DataFrame({'A': [10, 20], 'B': [30, 40]}, index=[0, 1])
df2 = pd.DataFrame({'A': [5, 5], 'B': [5, 5]}, index=[1, 2])
print("DF1:\n", df1)
print("\nDF2:\n", df2)
print("\nDF1 + DF2:")
print(df1 + df2)
```
**Output:**
```
DF1:
A B
0 10 30
1 20 40
DF2:
A B
1 5 5
2 5 5
DF1 + DF2:
A B
0 NaN NaN ← index 0 in DF1, but NOT in DF2 → NaN
1 25.0 45.0 ← index 1 in BOTH → added normally!
2 NaN NaN ← index 2 in DF2, but NOT in DF1 → NaN
```
**The `fill_value` trick — replace NaN with 0 during operation:**
```python
print(df1.add(df2, fill_value=0))
```
**Output:**
```
A B
0 10.0 30.0 ← 10 + 0 (fill_value fills missing df2 values with 0)
1 25.0 45.0
2 5.0 5.0 ← 0 + 5 (fill_value fills missing df1 values with 0)
```
**All Binary Operation Functions:**
| Operation | Symbol | Function | Example |
| :--- | :--- | :--- | :--- |
| Addition | + | `add()` | `df1.add(df2, fill_value=0)` |
| Subtraction | - | `sub()` | `df1.sub(df2, fill_value=0)` |
| Multiplication | * | `mul()` | `df1.mul(df2, fill_value=1)` |
| Division | / | `div()` | `df1.div(df2, fill_value=1)` |
> [!IMPORTANT]
> **Board Exam Tip**
> "What happens when you perform a binary operation between two DataFrames with different row indexes?" — **2-mark** question!
> Answer: Pandas aligns data by **both row index and column label**. Where matching labels exist, the operation is performed normally. Where labels exist in one DataFrame but not the other, the result is **NaN** (missing value).
---
## 2.4 📊 Descriptive Statistics with Pandas
**Descriptive Statistics** = functions that summarise your data — finding the average, highest, lowest, middle value, etc. Pandas makes all of this incredibly easy!
> **Analogy:** After a class test, a teacher wants to know: Who scored highest? What's the class average? What's the middle score? All of this is "descriptive statistics" — and Pandas answers all these questions in one line each!
**Using this DataFrame for all examples:**
```python
import pandas as pd
import numpy as np
df = pd.DataFrame({
'Maths': [90, 85, 78, 92, 88],
'Science': [88, 95, np.nan, 90, 82],
'English': [75, 80, 72, 85, 79]
}, index=['Arjun', 'Bina', 'Chirag', 'Divya', 'Esha'])
```
> **Remember:** All statistical functions **automatically skip NaN values** unless told otherwise!
---
### 2.4.1 Functions `min()` and `max()` 📉📈
```python
# min_max_demo.py
# Column-wise (default) — min/max of each subject
print("Lowest marks per subject:")
print(df.min())
print("\nHighest marks per subject:")
print(df.max())
# Row-wise (axis=1) — min/max per student
print("\nLowest marks per student:")
print(df.min(axis=1))
```
**Output:**
```
Lowest marks per subject:
Maths 78.0
Science 82.0
English 72.0
dtype: float64
Highest marks per subject:
Maths 92.0
Science 95.0
English 85.0
dtype: float64
Lowest marks per student:
Arjun 75.0
Bina 80.0
Chirag 72.0
Divya 85.0
Esha 79.0
dtype: float64
```
> `axis=0` (default) → calculates **down the column** (gives one value per column)
> `axis=1` → calculates **across the row** (gives one value per row)
---
### 2.4.2 Functions `mode()`, `mean()`, `median()` 📐
**Understanding the three "M"s:**
::: grid
::: card 📊 | Mean (Average) | Add all values ÷ count | (90+85+78+92+88)/5 = 86.6
::: card 🎯 | Median (Middle) | Sort and pick middle value | 78, 85, 88, 90, 92 → Middle = 88
::: card 🏆 | Mode (Most Common) | The value that appears most often | If 85 appears 3 times, mode = 85
:::
```python
# mean_median_mode.py
print("MEAN (Average marks per subject):")
print(df.mean())
print("\nMEDIAN (Middle value per subject):")
print(df.median())
print("\nMODE (Most common value per subject):")
print(df.mode())
```
**Output:**
```
MEAN (Average marks per subject):
Maths 86.6
Science 88.75
English 78.2
dtype: float64
MEDIAN (Middle value per subject):
Maths 88.0
Science 89.0
English 79.0
dtype: float64
MODE (Most common value per subject):
Maths Science English
0 78.0 82.0 72.0
```
> [!NOTE]
> **mode() returns a DataFrame, not a Series! 🧠**
> Because there can be MULTIPLE modes (e.g., if both 85 and 90 appear twice, both are modes!). That's why `mode()` returns a DataFrame with all possible mode values, while `mean()` and `median()` return a single Series.
---
### 2.4.3 Functions `count()` and `sum()` 🔢
```python
# count_sum_demo.py
print("COUNT (non-NaN values per subject):")
print(df.count())
print("\nSUM (total marks per subject):")
print(df.sum())
print("\nSUM per student (axis=1):")
print(df.sum(axis=1))
```
**Output:**
```
COUNT (non-NaN values per subject):
Maths 5
Science 4 ← Only 4! Chirag's NaN is not counted
English 5
dtype: int64
SUM (total marks per subject):
Maths 433.0
Science 355.0
English 391.0
dtype: float64
SUM per student (axis=1):
Arjun 253.0
Bina 260.0
Chirag 150.0 ← Only Maths + English (Science is NaN)
Divya 267.0
Esha 249.0
dtype: float64
```
> [!IMPORTANT]
> **Board Exam Tip**
> "What is the difference between count() and sum()?" — **1-mark** question!
> `count()` → returns the **number of non-NaN values** in each column/row.
> `sum()` → returns the **total (addition) of all non-NaN values** in each column/row.
---
### 2.4.4 Functions `quantile()`, `std()`, and `var()` 📉
These three sound scary, but let's explain them simply:
::: grid
::: card 📊 | std() — Standard Deviation | How spread out the marks are | Small std = everyone scored similarly; Large std = big differences
::: card 📐 | var() — Variance | std squared (std²) | Another measure of spread; variance = std × std
::: card 📏 | quantile() — Percentile | What value is at X% of the data | 0.25 = 25th percentile (25% of students scored below this)
:::
```python
# std_var_quantile_demo.py
print("STANDARD DEVIATION (how spread out marks are):")
print(df.std())
print("\nVARIANCE (std squared):")
print(df.var())
print("\n25th Percentile (Q1) of each subject:")
print(df.quantile(0.25))
print("\n50th Percentile (Median) of each subject:")
print(df.quantile(0.50))
print("\n75th Percentile (Q3) of each subject:")
print(df.quantile(0.75))
```
**Output:**
```
STANDARD DEVIATION:
Maths 5.177
Science 5.315
English 4.764
dtype: float64
VARIANCE:
Maths 26.80
Science 28.25
English 22.70
dtype: float64
25th Percentile (Q1):
Maths 85.0
Science 83.5
English 75.5
dtype: float64
50th Percentile (Median):
Maths 88.0
Science 89.0
English 79.0
dtype: float64
75th Percentile (Q3):
Maths 90.0
Science 91.25
English 80.5
dtype: float64
```
**What does Standard Deviation actually mean?**
```
Maths std = 5.177 → marks are close together (78 to 92 range)
If std were 30 → huge variation (some scored 10, some scored 100!)
```
> [!IMPORTANT]
> **Board Exam Tip**
> "What does `df.std()` return?" — **1-mark** question!
> Answer: `std()` returns the **Standard Deviation** of each column — it measures how much the values spread away from the mean. A small standard deviation means values are clustered close to the average; a large one means they are spread out widely.
---
### 2.4.5 The `describe()` Function — The Swiss Army Knife! 🛡️
`describe()` gives you **8 statistics at once** for all numeric columns — it's the most powerful summary function in Pandas!
```python
# describe_demo.py
print("Complete Statistical Summary:")
print(df.describe())
```
**Output:**
```
Maths Science English
count 5.000000 4.000000 5.000000
mean 86.600000 88.750000 78.200000
std 5.177370 5.315073 4.764452
min 78.000000 82.000000 72.000000
25% 85.000000 83.500000 75.500000
50% 88.000000 89.000000 79.000000
75% 90.000000 91.250000 80.500000
max 92.000000 95.000000 85.000000
```
**What each row means:**
| Row in describe() | Meaning |
| :--- | :--- |
| **count** | Number of non-NaN values |
| **mean** | Average of all values |
| **std** | Standard Deviation |
| **min** | Smallest value |
| **25%** | 25th percentile (Q1) |
| **50%** | Median (middle value) |
| **75%** | 75th percentile (Q3) |
| **max** | Largest value |
> [!IMPORTANT]
> **Board Exam Tip**
> "Which single function gives you count, mean, std, min, max of a DataFrame?" — **1-mark** question!
> Answer: **`df.describe()`** — it gives all 8 statistical summaries at once.
---
## 2.5 🔧 Some Other Essential Functions and Functionality
### 2.5.1 Inspection Function `info()` 🔍
`info()` gives you a **structural summary** of the DataFrame — not the data values, but information ABOUT the DataFrame.
```python
# info_demo.py
import pandas as pd
import numpy as np
df = pd.DataFrame({
'Name': ['Arjun', 'Bina', 'Chirag', 'Divya'],
'Maths': [90, 85, 78, 92],
'Science': [88, 95, np.nan, 90],
'Section': ['A', 'A', 'B', 'B']
})
df.info()
```
**Output:**
```
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 4 entries, 0 to 3
Data columns (total 4 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Name 4 non-null object
1 Maths 4 non-null int64
2 Science 3 non-null float64 ← Only 3 non-null (Chirag = NaN!)
3 Section 4 non-null object
dtypes: float64(1), int64(1), object(2)
memory usage: 256.0+ bytes
```
**What `info()` tells you:**
- Number of rows and columns
- Column names and their data types
- Count of non-NaN values per column
- Memory usage
> **Difference: `info()` vs `describe()`**
> `info()` → tells you ABOUT the structure (types, NaN counts, memory)
> `describe()` → tells you ABOUT the data (mean, max, std...)
---
### 2.5.2 DataFrame's Top and Bottom Rows — `head()` and `tail()` 📋
```python
# head_tail_demo.py
import pandas as pd
df = pd.DataFrame({
'Name': ['Arjun', 'Bina', 'Chirag', 'Divya', 'Esha', 'Farhan', 'Gita'],
'Marks': [90, 85, 78, 92, 88, 65, 95]
})
print("FIRST 3 rows (head):")
print(df.head(3))
print("\nLAST 3 rows (tail):")
print(df.tail(3))
print("\nDefault head (5 rows):")
print(df.head())
```
**Output:**
```
FIRST 3 rows (head):
Name Marks
0 Arjun 90
1 Bina 85
2 Chirag 78
LAST 3 rows (tail):
Name Marks
4 Esha 88
5 Farhan 65
6 Gita 95
Default head (5 rows):
Name Marks
0 Arjun 90
1 Bina 85
2 Chirag 78
3 Divya 92
4 Esha 88
```
| Function | Returns | Default |
| :--- | :--- | :--- |
| `df.head(n)` | First n rows | 5 |
| `df.tail(n)` | Last n rows | 5 |
---
### 2.5.3 Cumulative Calculation Functions 📈
**Cumulative** means adding up (or finding max/min) as you go — each row shows the running total up to that point.
> **Analogy:** Imagine counting your savings week by week:
> Week 1: ₹500 → Total so far: ₹500
> Week 2: ₹300 → Total so far: ₹800
> Week 3: ₹700 → Total so far: ₹1500
> That's exactly what `cumsum()` does!
```python
# cumulative_demo.py
import pandas as pd
sales = pd.DataFrame({
'Week': ['W1', 'W2', 'W3', 'W4', 'W5'],
'Sales': [500, 300, 700, 200, 600]
})
sales = sales.set_index('Week')
print("Original Sales:")
print(sales)
print("\nCumulative Sum (running total):")
print(sales.cumsum())
print("\nCumulative Max (highest seen so far):")
print(sales.cummax())
print("\nCumulative Min (lowest seen so far):")
print(sales.cummin())
print("\nCumulative Product:")
print(sales.cumprod())
```
**Output:**
```
Original Sales:
Sales
Week
W1 500
W2 300
W3 700
W4 200
W5 600
Cumulative Sum (running total):
Sales
Week
W1 500 ← Just W1: 500
W2 800 ← W1+W2: 500+300=800
W3 1500 ← W1+W2+W3: 800+700=1500
W4 1700 ← +200=1700
W5 2300 ← +600=2300
Cumulative Max (highest seen so far):
Sales
Week
W1 500
W2 500 ← Max(500,300) = 500
W3 700 ← Max(500,700) = 700
W4 700 ← Max(700,200) = 700 (no change)
W5 700 ← Max(700,600) = 700 (no change)
Cumulative Min (lowest seen so far):
Sales
Week
W1 500
W2 300 ← Min(500,300) = 300
W3 300 ← Min(300,700) = 300 (no change)
W4 200 ← Min(300,200) = 200
W5 200 ← Min(200,600) = 200 (no change)
```
**Four Cumulative Functions:**
| Function | What it does | Real use |
| :--- | :--- | :--- |
| `cumsum()` | Running total | Track total sales/savings |
| `cummax()` | Running maximum | Track record high score |
| `cummin()` | Running minimum | Track lowest temperature ever |
| `cumprod()` | Running product | Compound interest calculations |
> [!IMPORTANT]
> **Board Exam Tip**
> "What does cumsum() do?" — **2-mark** question.
> Answer: `cumsum()` returns a DataFrame where each value is the **cumulative sum** — the sum of all values from the beginning up to and including the current row. For example, if values are [10, 20, 30], cumsum gives [10, 30, 60].
---
### 2.5.4 Applying Functions on a Subset of DataFrame 🎯
You can apply a function to specific **rows or columns** using `apply()`:
```python
# apply_demo.py
import pandas as pd
df = pd.DataFrame({
'Maths': [90, 85, 78, 92, 88],
'Science': [88, 95, 70, 90, 82],
'English': [75, 80, 72, 85, 79]
}, index=['Arjun', 'Bina', 'Chirag', 'Divya', 'Esha'])
# Apply a function to each COLUMN (axis=0):
print("Max of each subject:")
print(df.apply(max)) # Applies Python's max() to each column
print("\nCustom function — range (max-min) per subject:")
print(df.apply(lambda x: x.max() - x.min()))
# Apply to each ROW (axis=1):
print("\nTotal marks per student:")
print(df.apply(sum, axis=1))
print("\nBest subject score per student:")
print(df.apply(max, axis=1))
```
**Output:**
```
Max of each subject:
Maths 92
Science 95
English 85
dtype: int64
Custom function — range (max-min) per subject:
Maths 14
Science 25
English 13
dtype: int64
Total marks per student:
Arjun 253
Bina 260
Chirag 220
Divya 267
Esha 249
dtype: int64
Best subject score per student:
Arjun 90
Bina 95
Chirag 78
Divya 92
Esha 88
dtype: int64
```
**Applying on a Subset of Columns:**
```python
# Only apply to Maths and Science columns:
subset = df[['Maths', 'Science']]
print("Mean of Maths and Science only:")
print(subset.mean())
```
> [!NOTE]
> **`apply()` is very powerful! 🧠**
> `apply()` lets you run ANY function across rows or columns — whether it's Python's built-in `max`, NumPy functions, or even your own custom `lambda` functions. It's the most flexible statistical tool in Pandas!
---
## 2.6 🧹 Handling Missing Data
**Missing data** (shown as `NaN`) is very common in real-world datasets:
- A student was absent for an exam
- A survey respondent skipped a question
- A sensor failed to record a reading
Pandas provides tools to **detect**, **remove**, and **fill** missing data.
```mermaid
graph TD
MISS["❓ Missing Data (NaN)"]
DET["🔍 Detect\nisnull() / notnull()"]
DROP["❌ Drop\ndropna()"]
FILL["✅ Fill\nfillna()"]
MISS --> DET
MISS --> DROP
MISS --> FILL
style MISS fill:#F44336,color:#fff
style DET fill:#FF9800,color:#fff
style DROP fill:#9E9E9E,color:#fff
style FILL fill:#4CAF50,color:#fff
```
**Our DataFrame with missing values:**
```python
import pandas as pd
import numpy as np
df = pd.DataFrame({
'Name': ['Arjun', 'Bina', 'Chirag', 'Divya', 'Esha'],
'Maths': [90, 85, np.nan, 92, 88],
'Science': [88, np.nan, 70, 90, np.nan],
'English': [75, 80, 72, 85, 79]
})
print(df)
```
**Output:**
```
Name Maths Science English
0 Arjun 90.0 88.0 75
1 Bina 85.0 NaN 80
2 Chirag NaN 70.0 72
3 Divya 92.0 90.0 85
4 Esha 88.0 NaN 79
```
---
### 2.6.1 Detecting / Filtering Missing Data 🔍
```python
# detect_missing.py
# isnull() — returns True where data is MISSING
print("Where is data missing?")
print(df.isnull())
# notnull() — returns True where data IS PRESENT
print("\nWhere is data present?")
print(df.notnull())
# Count missing values per column:
print("\nCount of missing values per column:")
print(df.isnull().sum())
# Which rows have ANY missing value?
print("\nRows with at least one missing value:")
print(df[df.isnull().any(axis=1)])
```
**Output:**
```
Where is data missing?
Name Maths Science English
0 False False False False
1 False False True False
2 False True False False
3 False False False False
4 False False True False
Count of missing values per column:
Name 0
Maths 1
Science 2
English 0
dtype: int64
Rows with at least one missing value:
Name Maths Science English
1 Bina 85.0 NaN 80
2 Chirag NaN 70.0 72
4 Esha 88.0 NaN 79
```
---
### 2.6.2 Handling Missing Data — Dropping Missing Values ❌
`dropna()` removes rows (or columns) that contain NaN values:
```python
# dropna_demo.py
print("Original DataFrame:")
print(df[['Name', 'Maths', 'Science']])
print("\nAfter dropna() — removes rows with ANY NaN:")
print(df.dropna()[['Name', 'Maths', 'Science']])
print("\nAfter dropna(how='all') — removes rows where ALL values are NaN:")
print(df.dropna(how='all')[['Name', 'Maths', 'Science']])
# Drop columns with NaN:
print("\nDrop COLUMNS with NaN (axis=1):")
print(df.dropna(axis=1))
# Drop rows where specific column has NaN:
print("\nDrop rows where Maths is NaN:")
print(df.dropna(subset=['Maths']))
```
**Output:**
```
Original DataFrame:
Name Maths Science
0 Arjun 90.0 88.0
1 Bina 85.0 NaN
2 Chirag NaN 70.0
3 Divya 92.0 90.0
4 Esha 88.0 NaN
After dropna() — removes rows with ANY NaN:
Name Maths Science
0 Arjun 90.0 88.0
3 Divya 92.0 90.0
After dropna(how='all'):
(same as original — no row where ALL are NaN)
Drop COLUMNS with NaN (axis=1):
Name English
0 Arjun 75
1 Bina 80
2 Chirag 72
3 Divya 85
4 Esha 79
Drop rows where Maths is NaN:
Name Maths Science English
0 Arjun 90.0 88.0 75
1 Bina 85.0 NaN 80
3 Divya 92.0 90.0 85
4 Esha 88.0 NaN 79
```
**`dropna()` Parameters:**
| Parameter | Default | Meaning |
| :--- | :--- | :--- |
| `axis` | 0 | 0=drop rows, 1=drop columns |
| `how` | 'any' | 'any'=drop if ANY NaN; 'all'=drop only if ALL are NaN |
| `subset` | None | Only check specific columns for NaN |
| `inplace` | False | True = modify original; False = return new df |
---
### 2.6.3 Handling Missing Data — Filling Missing Values ✅
Instead of deleting rows with NaN, you can **replace NaN with a meaningful value** using `fillna()`:
```python
# fillna_demo.py
import pandas as pd
import numpy as np
df = pd.DataFrame({
'Name': ['Arjun', 'Bina', 'Chirag', 'Divya', 'Esha'],
'Maths': [90, 85, np.nan, 92, 88],
'Science': [88, np.nan, 70, 90, np.nan],
})
print("Original:")
print(df)
# Fill ALL NaN with a fixed value:
print("\nFill NaN with 0:")
print(df.fillna(0))
# Fill NaN with the COLUMN MEAN (most common approach):
print("\nFill NaN with column mean:")
print(df.fillna(df.mean()))
# Fill NaN forward (use previous row's value):
print("\nForward fill (ffill):")
print(df.fillna(method='ffill'))
# Fill NaN backward (use next row's value):
print("\nBackward fill (bfill):")
print(df.fillna(method='bfill'))
# Fill specific columns with specific values:
print("\nFill Maths NaN with 0, Science NaN with 60:")
print(df.fillna({'Maths': 0, 'Science': 60}))
```
**Output:**
```
Original:
Name Maths Science
0 Arjun 90.0 88.0
1 Bina 85.0 NaN
2 Chirag NaN 70.0
3 Divya 92.0 90.0
4 Esha 88.0 NaN
Fill NaN with 0:
Name Maths Science
0 Arjun 90.0 88.0
1 Bina 85.0 0.0 ← NaN replaced with 0
2 Chirag 0.0 70.0 ← NaN replaced with 0
3 Divya 92.0 90.0
4 Esha 88.0 0.0 ← NaN replaced with 0
Fill NaN with column mean:
Name Maths Science
0 Arjun 90.0 88.0
1 Bina 85.0 82.67 ← Mean of (88,70,90) = 82.67
2 Chirag 88.75 70.0 ← Mean of (90,85,92,88) = 88.75
3 Divya 92.0 90.0
4 Esha 88.0 82.67 ← Mean again
Forward fill:
Name Maths Science
0 Arjun 90.0 88.0
1 Bina 85.0 88.0 ← Takes value from row above (88)
2 Chirag 85.0 70.0 ← Takes Maths from row above (85)
3 Divya 92.0 90.0
4 Esha 88.0 90.0 ← Takes value from row above (90)
```
**`fillna()` Strategies — When to use which:**
| Strategy | Code | When to use |
| :--- | :--- | :--- |
| Fixed value | `fillna(0)` | Zero means "not attempted" |
| Column mean | `fillna(df.mean())` | Best estimate for numeric data |
| Column median | `fillna(df.median())` | When data has outliers |
| Forward fill | `fillna(method='ffill')` | Time-series data (carry last known value) |
| Backward fill | `fillna(method='bfill')` | Use next known value |
> [!IMPORTANT]
> **Board Exam Tip**
> "Write code to fill all NaN values in a DataFrame with the column's mean." — **2-mark** question!
> Answer: `df.fillna(df.mean())`
>
> "Write code to drop all rows that have any missing value." — **2-mark** question!
> Answer: `df.dropna()` or `df.dropna(how='any')`
---
## 2.7 📦 Function `groupby()` — Group and Analyse!
`groupby()` is one of the most powerful functions in Pandas. It lets you:
1. **Split** data into groups based on a column
2. **Apply** a function (sum, mean, count, etc.) to each group
3. **Combine** results into a new DataFrame
> **Super Simple Analogy:** Imagine dividing your class into **Section A and Section B** and then finding the average marks of each section separately. That's exactly what `groupby('Section').mean()` does!
```mermaid
graph LR
ALL["📋 All Students\n(Mixed Sections)"]
GRP["groupby('Section')"]
A["📊 Section A\nArjun, Bina, Esha"]
B["📊 Section B\nChirag, Divya"]
AGG["apply mean()"]
RES["📊 Result\nA: avg marks\nB: avg marks"]
ALL --> GRP
GRP --> A
GRP --> B
A --> AGG
B --> AGG
AGG --> RES
style GRP fill:#FF9800,color:#fff
style RES fill:#4CAF50,color:#fff
```
**Basic `groupby()` example:**
```python
# groupby_basic.py
import pandas as pd
import numpy as np
df = pd.DataFrame({
'Name': ['Arjun', 'Bina', 'Chirag', 'Divya', 'Esha', 'Farhan'],
'Section': ['A', 'A', 'B', 'B', 'A', 'B'],
'Maths': [90, 85, 78, 92, 88, 74],
'English': [75, 80, 72, 85, 79, 68]
})
print("Original DataFrame:")
print(df)
# Group by Section and find MEAN of each group:
print("\nSection-wise AVERAGE:")
print(df.groupby('Section').mean())
# Group by Section and find MAX of each group:
print("\nSection-wise MAXIMUM:")
print(df.groupby('Section').max())
# Group by Section and find COUNT:
print("\nSection-wise COUNT (number of students):")
print(df.groupby('Section').count())
```
**Output:**
```
Original DataFrame:
Name Section Maths English
0 Arjun A 90 75
1 Bina A 85 80
2 Chirag B 78 72
3 Divya B 92 85
4 Esha A 88 79
5 Farhan B 74 68
Section-wise AVERAGE:
Maths English
Section
A 87.67 78.00
B 81.33 75.00
Section-wise MAXIMUM:
Name Maths English
Section
A Esha 90 80
B Farhan 92 85
Section-wise COUNT:
Name Maths English
Section
A 3 3 3
B 3 3 3
```
**More `groupby()` Examples:**
```python
# groupby_advanced.py
# Sum of marks per section:
print("Total marks per section:")
print(df.groupby('Section')[['Maths', 'English']].sum())
# Groupby and get specific statistic:
print("\nMinimum Maths marks per section:")
print(df.groupby('Section')['Maths'].min())
# Get complete statistics per group:
print("\nFull statistics per section:")
print(df.groupby('Section').describe())
```
**Output:**
```
Total marks per section:
Maths English
Section
A 263 234
B 244 225
Minimum Maths marks per section:
Section
A 85
B 74
Name: Maths, dtype: int64
```
**Groupby with multiple columns:**
```python
# Add City column to demo multiple groupby:
df['City'] = ['Delhi', 'Mumbai', 'Delhi', 'Pune', 'Mumbai', 'Delhi']
print("Section AND City-wise average:")
print(df.groupby(['Section', 'City'])['Maths'].mean())
```
> [!IMPORTANT]
> **Board Exam Tip**
> "Write code to find the average marks of each section using groupby()." — **3-mark** question, extremely common!
> Answer:
> ```python
> result = df.groupby('Section').mean()
> print(result)
> ```
---
## ⚠️ Common Errors and Misconceptions
| Mistake | What Goes Wrong | Correct Approach |
| :--- | :--- | :--- |
| ❌ Using `and` instead of `&` in conditions | TypeError | ✅ `df[(df['M']>80) & (df['S']=='A')]` |
| ❌ Forgetting NaN skipping behaviour | Think NaN = 0 | ✅ NaN is SKIPPED in mean(), sum(), etc. |
| ❌ `iteritems()` in new Pandas | DeprecationWarning/Error | ✅ Use `items()` in Pandas 1.5+ |
| ❌ `dropna()` not changing original | df unchanged | ✅ Use `inplace=True` or `df = df.dropna()` |
| ❌ Confusing `count()` and `size()` | Wrong answer | ✅ `count()` skips NaN; `size` includes NaN |
| ❌ mode() returning a Series | Confused by DataFrame output | ✅ mode() returns a DataFrame (multiple modes possible!) |
---
## 🔑 Quick Revision — Exam Ready!
**Iteration:**
- `iterrows()` → row by row → `(index, Series)`
- `items()` / `iteritems()` → column by column → `(col_name, Series)`
**Statistical Functions:**
| Function | Returns |
| :--- | :--- |
| `min()` / `max()` | Smallest/largest per column |
| `mean()` | Average per column |
| `median()` | Middle value per column |
| `mode()` | Most common value (DataFrame) |
| `sum()` | Total per column |
| `count()` | Non-NaN count per column |
| `std()` | Standard Deviation |
| `var()` | Variance (std²) |
| `quantile(q)` | Value at q-th percentile |
| `describe()` | All 8 stats at once |
**Missing Data:**
- `isnull()` → True where NaN | `notnull()` → True where not NaN
- `dropna()` → remove rows/columns with NaN
- `fillna(value)` → replace NaN with value
**groupby():**
- `df.groupby('col').mean()` → group by col, find average
- `df.groupby('col').sum()` → group by col, find total
- `df.groupby('col').count()` → count per group
**Cumulative:**
- `cumsum()` → running total
- `cummax()` → running maximum
- `cummin()` → running minimum
---
## 🎯 Sample Board Exam Questions
### Q1: Very Short Answer [1 mark each]
a) Which function gives all 8 statistics (count, mean, std, min, 25%, 50%, 75%, max) in one call?
**→ `df.describe()`**
b) What does `isnull()` return?
**→ A DataFrame of True/False — True where values are missing (NaN), False where present.**
c) What does `cumsum()` do?
**→ Returns the cumulative running sum — each value is the sum of all elements up to that position.**
d) Which function removes rows containing NaN?
**→ `dropna()`**
e) What is the difference between `count()` and `sum()`?
**→ `count()` returns the number of non-NaN values; `sum()` returns the total (addition) of all non-NaN values.**
---
### Q2: Short Answer [2 marks]
**Q: Differentiate between `dropna()` and `fillna()`.**
`dropna()` **removes** rows (or columns) that contain NaN values — the affected rows are deleted from the result.
`fillna(value)` **replaces** NaN values with a specified value (e.g., 0, the column mean, or 'Unknown') — no rows are removed.
```python
df.dropna() # Removes Chirag's row (NaN in Science)
df.fillna(0) # Keeps Chirag but replaces his NaN with 0
```
---
### Q3: Program Writing [3 marks]
**Q: Create a DataFrame of 4 students with Maths and Science marks (one NaN). Find section-wise average using groupby(). Also fill NaN with the column mean.**
```python
import pandas as pd
import numpy as np
data = {
'Name': ['Arjun', 'Bina', 'Chirag', 'Divya'],
'Section': ['A', 'A', 'B', 'B'],
'Maths': [90, 85, np.nan, 92],
'Science': [88, 95, 70, 90]
}
df = pd.DataFrame(data)
print("Original:")
print(df)
# Fill NaN with mean:
df_filled = df.fillna(df.mean())
print("\nAfter filling NaN with mean:")
print(df_filled)
# Section-wise average:
print("\nSection-wise average:")
print(df.groupby('Section').mean())
```
---
### Q4: Output Based [3 marks]
**Q: What is the output?**
```python
import pandas as pd
import numpy as np
s = pd.Series([10, 20, 30, 40, 50])
print(s.cumsum())
print(s.std())
```
**Output:**
```
0 10
1 30
2 60
3 100
4 150
dtype: int64
15.811388300841896
```
*(cumsum: running totals 10, 10+20=30, 30+30=60, 60+40=100, 100+50=150)*
*(std: standard deviation of [10,20,30,40,50])*
---
## ✏️ Practice Problems
1. Create a DataFrame of 5 products with their prices and quantities. Use `describe()` to get the statistical summary.
2. Write code to iterate over each row of a student DataFrame and print "Pass" if marks ≥ 40, else "Fail".
3. Create two DataFrames with matching and non-matching indexes. Add them using `.add(fill_value=0)` and explain the result.
4. From a DataFrame with NaN values, write code to: (a) count NaN per column, (b) drop rows with NaN, (c) fill NaN with the column median.
5. Create a DataFrame with students from different cities. Use `groupby('City')` to find the maximum marks per city.
6. Find the cumulative sum of a Series [5, 10, 15, 20] and predict the output before running.
7. Write code to apply a custom function that calculates (max - min) for each column of a DataFrame.
8. What is the difference between `iterrows()` and `items()`? Write one example of each.
9. A DataFrame has columns 'Month' and 'Sales'. Use `cumsum()` to show running total sales and `cummax()` to track the best month so far.
10. Create a DataFrame, then: (a) find mode of each column, (b) find the 75th percentile (Q3) using quantile(), (c) find standard deviation.
Back to List
Calculating...
UNIT 1 : CH 2
Dec 14, 2025
📊 Data Handling Using Pandas – II
Learning Support
Need Help With This Chapter?
Save key topics for exam revision, ask questions to teachers, or submit content corrections.
Verified Doubts & Teacher Answers
Loading resolved questions for this note...