Skip to content

Accessing, Modifying and Filtering Data

After loading a dataset, you'll often need to access specific rows, extract particular columns, remove data you don't need, replace certain values, or create entirely new fields. These operations are central to preparing a dataset for serious analysis, raw data is rarely ready to use exactly as it arrives.

This part focuses on indexing, column selection, dropping rows or columns, replacing values, inserting new fields, and filtering observations that meet specific conditions.

Python Code Examples: Accessing and Modifying Data

A quick reminder of the data

data.head(5)

Dropping rows

# Delete the first and third passengers
# Remember: indexing in Python starts from 0, so these are at positions 0 and 2
data.drop([0, 2], axis=0).head(5)

Dropping columns

# You can also delete entire columns, not just rows
# For example, the 'Cabin' column might not be relevant to your analysis, so let's remove it
data.drop(['Cabin'], axis=1).head(5)

An important detail: .drop() doesn't modify the original data

data.head(5)

Notice that data still contains the Cabin column here, even after the .drop() call above. That's because, by default, .drop() returns a new DataFrame with the change applied, rather than altering the original in place. This is a deliberate safety feature in pandas: it stops you from accidentally destroying your original data with a single careless line of code.

# Confirming this: both the original data and the result of .drop() are DataFrames,
# but they are two separate objects
print(type(data))
print(type(data.drop(['Cabin'], axis=1)))

If you do want to permanently apply a change like this, you'd either reassign the result back to a variable (e.g. data = data.drop(['Cabin'], axis=1)), or pass the argument inplace=True to the method itself.

Practice: applying a drop and keeping the result

# Compute the reduced data matrix by dropping the 'Cabin' column
reduced_data = data.drop(['Cabin'], axis=1)

# Printing the variable displays it directly, useful for a quick check
reduced_data.head()

Replacing Values

# .replace() swaps every matching value in the DataFrame for a new one
# Here, every occurrence of 'male' is replaced with the number 1
data.replace('male', 1).head(5)

# A more targeted and commonly preferred approach: apply .map() to a single column
# This converts the categorical text values in 'Sex' into numeric codes,
# which is often necessary before feeding data into statistical or machine learning models
data['Sex'] = data['Sex'].map({'female': 1, 'male': 0})

Why use .map() on a single column rather than .replace() on the whole DataFrame? .replace() searches the entire DataFrame for matching values, which can accidentally affect columns you didn't intend to change (imagine another column that happened to also contain the text "male"). Targeting a specific column with .map() is generally the safer, more deliberate choice.

data.head(5)

Inserting a New Column

# .insert() adds a new column at a specific position
# Here, we insert a column called 'Nationality' at position 5 (right after 'Sex')
# and give every row the same default value, 'Irish', to start with
data.insert(5, 'Nationality', 'Irish')

data.head(5)

Refining the new column based on another column's values

# Suppose we assume that passengers with a missing (NaN) Age are, in fact,
# of a different nationality, 'European', perhaps explaining why their age was never recorded

# Don't worry about exactly how np.isnan() works yet, it will make more sense
# once we've properly covered NumPy
import numpy as np

for index, row in data.iterrows():
    if np.isnan(data.loc[index, "Age"]):
        data.loc[index, "Nationality"] = "European"

data.head(10)

Understanding .loc

The example above relies on .loc, one of the most important tools in pandas for accessing and modifying specific rows and columns by their labels.

# .loc[3] selects the entire row labelled 3 (by default, this matches its position too)
print(data.loc[3])

# .loc offers a great deal of additional functionality beyond simple row selection
# You can explore this further by running: data.loc?? (in a Jupyter Notebook)
# or by checking Python's built-in help function:
help(data.loc)

Filtering with Boolean Conditions

# Comparing a column to a value creates a Series of True/False values, one per row
european_bool = data['Nationality'] == 'European'
print(european_bool.head(10))

# Using this Boolean Series to index into the DataFrame keeps only the matching rows
europeans = data[european_bool]

# .value_counts() tallies how many times each unique value appears in a column
data['Nationality'].value_counts()

# Counting only the non-missing (non-null) entries in a column can be done with .count()
data['Nationality'].count()

A pattern worth remembering: data[condition] is one of the most frequently used patterns in all of pandas. Whenever you want to filter a dataset down to rows meeting some criteria, whether that's a specific nationality, an age range, or a passenger class, the general recipe is the same: build a Boolean condition first (as its own variable, for clarity), then use it to index into the DataFrame.

E.4 Practice and Titanic Extension

This part provides additional practice using the Titanic dataset. The aim is to reinforce your core pandas skills by asking you to compute values, subset data, and apply the tools introduced above to a more realistic, messier dataset than the toy examples used earlier.

Treat these examples as an opportunity to experiment. Run the code, inspect each output carefully, and try adjusting small parts of the commands, changing a condition, a column name, or a value, to test and deepen your understanding.

Python Code Examples: Optional Practice

Practice: counting values that meet a condition

# Compute the number of people with European nationality
n_european = (data['Nationality'] == 'European').sum()

# Compute the number of people for whom we have cabin information
# .notna() flags entries that are NOT missing; summing a Boolean Series counts the True values
n_passengers_cabin = data['Cabin'].notna().sum()

print(n_european, n_passengers_cabin)

Why .sum() works here: in Python, True behaves like 1 and False behaves like 0 when used in arithmetic. So summing a column of True/False values conveniently gives you a straightforward count of how many entries were True, a small but very useful trick that comes up constantly in pandas.

Practice: filtering rows by a condition

# Create a new DataFrame containing only the passengers with Pclass == 2
# One common way to apply a condition in pandas is directly as data[CONDITION]
only_pclass2 = data[data['Pclass'] == 2]

# Printing the first few rows lets us quickly check the filter worked as expected
only_pclass2.head()

Loading additional related datasets

import pandas as pd

# Reload the original Titanic dataset
data_org = pd.read_csv('data/titanic.csv')

# Load a second, related dataset containing nationality information
data_nat = pd.read_csv('data/nationalities.csv')

Having two related datasets like this sets up a natural next step, combining or merging them together based on a shared column (such as a passenger ID), which is a common and important pandas skill covered in the next section.