Skip to content

Pandas

This section introduces pandas, one of the main Python libraries for working with structured datasets. Pandas is especially useful whenever data is organised in tables, such as CSV files, spreadsheets, financial datasets, or survey data, essentially, anywhere you'd naturally think in terms of rows and columns.

The material is organised into a short sequence: first, we introduce DataFrames using simple toy data; then we load and inspect a real dataset; after that, we access, modify, and filter data; finally, we practise these skills and introduce some more advanced DataFrame operations.

Learning Focus

By the end of this section, you should be able to:

  • Explain the purpose of pandas in Python data analysis.
  • Create and inspect basic DataFrames.
  • Load a CSV file into pandas.
  • Access rows, columns, and summary information.
  • Modify, filter, and transform tabular data.
  • Recognise how pandas supports more advanced empirical analysis.

E.1 Pandas and DataFrames

Pandas provides data analysis and statistical functionality in Python. It's designed specifically for working with structured data, especially data arranged neatly in rows and columns.

The central object you'll work with is the DataFrame. It can be thought of as a spreadsheet-like table: it has rows, columns, column names, and values, all of which can be inspected, summarised, and transformed using pandas' built-in tools.

Python Code Examples: Creating and Inspecting DataFrames

Generating some toy data to work with

import pandas as pd
import numpy as np

# We fix the random seed so the "random" data generated below is reproducible
# Please do not change this seed if you're following along with the same expected results
np.random.seed(seed=9)

# Generate some toy data: 100 rows, 1 column, sampled from a standard normal (Gaussian) distribution
values = np.random.randn(100, 1)

Setting a fixed seed, just as we saw earlier with NumPy, means that even though the values are generated "randomly," everyone running this exact code will get the exact same numbers, which is essential when following worked examples or comparing results with others.

Creating a DataFrame from an array

# Use the data generated above to create a DataFrame
dataframe = pd.DataFrame(data=values)

Inspecting a DataFrame

# .shape gives the dimensions of the DataFrame: (number of rows, number of columns)
print(dataframe.shape)     # (100, 1)

# Many pandas methods share names with familiar Python or NumPy functions,
# such as .max(), .min(), and .sum(). Others include .median(), .mean(), and .std()
print(dataframe.max())

# Note: the result isn't a single plain number, it's returned as a pandas "Series"
print('Result type:', type(dataframe.max()))   # <class 'pandas.core.series.Series'>

# To extract just the single maximum value itself, without the extra Series wrapper,
# you can index into the result using its column position
print(dataframe.max()[0])

A note on why this matters: when a DataFrame has only one column, it's easy to forget that pandas still returns results as a Series (a labelled, one-dimensional structure) rather than a plain number. Getting comfortable with this distinction early on will save confusion later, especially once your DataFrames have multiple columns and .max() naturally returns one value per column.

Practice: extracting single summary values

# Fill in the line below so that min_elem_df contains ONLY the minimum value of dataframe
min_elem_df = dataframe.min()[0]

# Fill in the line below so that mean_elem_df contains ONLY the mean value of dataframe
mean_elem_df = dataframe.mean()[0]

print(min_elem_df, mean_elem_df)

Practice: working with a larger sample

# Resetting the seed again ensures this next dataset is also reproducible
np.random.seed(seed=9)

# Create a DataFrame containing 10,000 randomly generated values instead of 100
dataframe_10k = pd.DataFrame(data=np.random.randn(10000, 1))

# Compute the mean of this larger dataset
mean_10k = dataframe_10k.mean()[0]
print(mean_10k)

Note: with a much larger sample (10,000 values rather than 100), you'll notice the mean sits even closer to 0, this is a nice, concrete illustration of the law of large numbers: as a random sample grows, its average tends to settle closer and closer to the true underlying mean of the distribution it was drawn from.

Creating a DataFrame from a Dictionary

Toy numerical data is useful for practice, but real datasets are usually more meaningful when they combine multiple related columns. Dictionaries are a natural and common way to construct this kind of DataFrame directly in code:

# Build a dictionary where each key becomes a column, and each value is a list of entries
d1 = {}
d1['countries'] = ['UK', 'France', 'Spain', 'Netherlands']
d1['codes'] = ['uk', 'fr', 'es', 'nl']

# Population is measured in millions
d1['population'] = [65.6, 66.9, 46.6, 17.0]

# GDP is measured in billions
d1['gdp'] = [2619, 2465, 1232, 770]

# DataFrame's constructor accepts a dictionary directly, turning each key into a column
countries_data = pd.DataFrame(d1)

print(countries_data['gdp'])   # accessing a single column by its name
countries_data                  # in a notebook, this displays a nicely formatted HTML table

Summarising numeric columns only

# Calling an aggregation method like .mean() on the whole DataFrame gives one result per column,
# which makes sense: we wouldn't want to average populations together with GDP figures!
# numeric_only=True ensures pandas skips non-numeric columns (like 'countries' and 'codes')
countries_data.mean(numeric_only=True)

Practice: summarising specific columns

# Fill in the line below so that sum_pop_countries_data computes the sum of the 
# population column, rounded to one decimal place
sum_pop_countries_data = round(countries_data['population'].sum(), 1)

# Fill in the line below so that std_gdp_countries_data computes the standard deviation
# of the gdp column, rounded to one decimal place
std_gdp_countries_data = round(countries_data['gdp'].std(), 1)

print(sum_pop_countries_data, std_gdp_countries_data)

A tip on structuring exercises like this: when a task asks you to compute a single, specific summary value, it's good practice to chain methods directly, select the column first (countries_data['population']), then apply the summary method (.sum()), and finally apply any rounding (round(..., 1)), reading the line from left to right as a sequence of steps applied to the data.

E.2 Loading and Inspecting Data

Once you're comfortable with the basic DataFrame structure, the natural next step is learning to load data from an external file. In practice, the vast majority of real data analysis begins this way: importing a dataset and immediately checking its structure before doing anything else with it.

This part introduces loading a CSV file, inspecting its rows and columns, viewing summary statistics, and producing a simple plot directly from pandas.

Python Code Examples: Loading and Inspecting Data

Loading a CSV file

# pd.read_csv() reads a CSV file from disk into a pandas DataFrame
data = pd.read_csv('data/titanic.csv')

# Checking the shape tells us immediately how many rows and columns we've loaded
print(data.shape)

Exploring a newly loaded dataset

The very first step in any data analysis project is exploration: confirming that the dataset loaded correctly, and getting an initial feel for what it actually contains.

# .head(5) displays the first 5 rows, a quick way to sanity-check the data
data.head(5)
# (When run inside a Jupyter Notebook, this renders as a neatly formatted HTML table.)

# .describe() generates summary statistics (count, mean, std, min, max, and quartiles)
# for every numeric column in the DataFrame, all in one call
data.describe()

Visualising a column directly from pandas

# Pandas can generate plots directly, using matplotlib behind the scenes
# (matplotlib must be installed in your environment for this to work)

# Plot a histogram showing the distribution of passenger ages
data.hist(column='Age')

This is often one of the fastest ways to spot issues in a new dataset, for example, an unexpected spike at age 0 might suggest missing ages were filled in with a placeholder value, rather than genuinely representing infants.

Practice: accessing individual values

# Fill in the line below so that age_passenger_10 contains the age of the 10th passenger
# Remember: pandas uses zero-based indexing, so the 10th passenger is at index 9
age_passenger_10 = data['Age'][9]

# Fill in the line below so that cabin_194_passenger contains the cabin number
# of the 194th passenger (index 193)
cabin_194_passenger = data['Cabin'][193]

print(age_passenger_10, cabin_194_passenger)

Practice: accessing another column

# Fill in the line below so that it computes the ticket number/ID of the 100th passenger (index 99)
ticket_i_th_passenger = data['Ticket'][99]

# Simply referencing the variable on its own line displays its value
# when run in a notebook cell (you don't need to change this)
ticket_i_th_passenger

A note on indexing real datasets: as with strings and lists earlier in this course, remember that pandas indexing starts at 0. So "the 10th passenger" corresponds to index 9, and "the 194th passenger" corresponds to index 193. This off-by-one distinction is one of the most common early sources of subtly wrong results, so it's always worth double-checking exactly which row you intend to select.