Data Inspection

Before building models or drawing conclusions, analysts should always explore their data first. Exploratory analysis helps identify whether a dataset behaves as expected, whether variables contain missing values, whether extreme observations exist, and whether different groups within the data behave differently from one another.

In business analysis, descriptive statistics often provide the very first view into how a variable behaves. The mean describes average performance, the standard deviation gives a basic measure of volatility, and the minimum and maximum help identify unusually large movements. Grouped statistics then allow comparison across categories, for example, positive versus negative return days, different industries, countries, or management groups.

E.1 Descriptive and Grouped Statistics

This subsection combines several complementary forms of data exploration. The command describe() gives a fast statistical overview of an entire dataset in one call. The method groupby() allows the same statistic to be calculated separately across distinct groups. Validation checks such as isna() and simple row counts help confirm whether the data is genuinely suitable for the analysis you're about to build on top of it.

Python Code Examples: Exploration and Validation

# Descriptive statistics
summary = sp500.describe()
print(summary)

# Select only return-related variables
return_summary = sp500[["return_sp500", "return_percent", "rolling_volatility_5d"]].describe()
print(return_summary)

# Grouped analysis: compare positive and non-positive return days
grouped = sp500.groupby("positive_return")

print(grouped["return_sp500"].mean())
print(grouped["return_sp500"].std())
print(grouped["return_sp500"].count())

# A more compact grouped summary, all in one call
group_table = grouped["return_sp500"].agg(["count", "mean", "std", "min", "max"])
print(group_table)

# Validation checks
print("Number of rows:", len(sp500))
print("Missing values:")
print(sp500.isna().sum())

Expected output / what to inspect

Inspect the summary statistics, the grouped table, the total row count, and the missing-value checks.

Making sense of each step:

.describe() produces, for every numeric column at once, the count of non-missing values, the mean, standard deviation, minimum, maximum, and the 25th/50th/75th percentiles. Running this immediately after loading or preparing a dataset is one of the fastest ways to catch problems early, for example, a minimum value that's implausibly negative, or a maximum wildly larger than every other observation, both worth investigating before trusting any further analysis.
Narrowing .describe() to specific columns (sp500[[...]].describe()) is a useful refinement once a dataset has many columns of different kinds (prices, returns, indicators), since it avoids a cluttered summary mixing variables that aren't meaningfully comparable.
.groupby("positive_return") splits the dataset into separate groups based on that column's values (here, simply 0 and 1), without needing to write a manual loop or filter each group by hand. This is precisely the "split-apply-combine" pattern introduced in Session 2.
Comparing .mean(), .std(), and .count() across the two groups answers a genuinely interesting question: do positive-return days and negative-return days differ systematically in magnitude or in how often they occur? Financial theory would suggest some asymmetry here is common and worth investigating further.
.agg([...]) is a convenient shortcut for computing several grouped statistics in a single line, rather than calling .mean(), .std(), .min(), and .max() separately and manually stitching the results together.
The final validation checks — row count and missing-value counts — act as a last sanity check before moving on to more advanced work like merging (Section F). It's good practice to re-run checks like these any time a dataset has been filtered, transformed, or reloaded, since it's easy to accidentally introduce a subtle problem partway through a longer workflow.

Why exploration comes before modelling: it can be tempting to jump straight into building a model or drawing a conclusion once a dataset is loaded. But a few minutes spent on .describe(), .groupby(), and .isna() routinely surfaces exactly the kind of hidden data issues, missing periods, duplicated values, unexpected outliers, that would otherwise quietly undermine everything built on top of them later.