Skip to content

Data Analysis

Many of the most important variables in business analysis are not directly observed, they're calculated from raw data. For example, analysts usually work with returns (percentage changes) rather than raw price levels, since returns are far easier to compare across different assets with very different price levels. Similarly, indicator variables can represent business conditions, such as whether a return was positive or negative on a given day.

Creating new variables like these is therefore a core part of applied analysis. In management datasets, this might involve constructing performance-change measures, customer categories, or risk indicators. In economics, it might involve growth rates, inflation-adjusted values, or indicators marking specific policy periods.

D.1 Returns, Indicators, Lags and Rolling Measures

This subsection introduces several common transformations you'll rely on constantly. The method pct_change() calculates percentage changes, which is exactly how returns are computed from a series of prices. The function np.select() creates indicator variables from one or more conditions. The methods shift() and rolling() are useful for creating lagged variables and moving-window calculations, respectively.

These operations are widely used in financial modelling, where analysts routinely need returns, lagged returns, rolling volatility, and indicators marking different market states.

Python Code Examples: Creating Business Variables

import numpy as np

# Calculate arithmetic returns from prices
sp500["return_sp500"] = sp500["sp500_price"].pct_change()

# Remove the first missing return
# (the very first row has no prior price to compare against, so pct_change() leaves it as NaN)
sp500.dropna(subset=["return_sp500"], inplace=True)

# Create an indicator variable:
# 1 if the return is positive, 0 otherwise
conditions = [
    sp500["return_sp500"] > 0,
    sp500["return_sp500"] <= 0
]
values = [1, 0]

sp500["positive_return"] = np.select(conditions, values)

# Additional useful transformations
sp500["return_percent"] = sp500["return_sp500"] * 100
sp500["lag_return"] = sp500["return_sp500"].shift(1)
sp500["rolling_volatility_5d"] = sp500["return_sp500"].rolling(window=5).std()

print(sp500.head(10))

Expected output / what to inspect

Inspect return_sp500, positive_return, lag_return, and rolling_volatility_5d.

Unpacking each transformation:

  • .pct_change() computes, for each row, the percentage change relative to the previous row: (current − previous) / previous. This is precisely the standard definition of a simple return in finance. Because the very first observation has no earlier row to compare against, its return is automatically NaN, hence the .dropna() step immediately afterward to remove that single unusable row.
  • np.select(conditions, values) works through the list of conditions in order and assigns the corresponding value from values the first time a condition is met for each row. Here, with only two mutually exclusive conditions (> 0 and <= 0), it behaves like a compact if/else applied across the entire column at once, avoiding a manual loop entirely. np.select() becomes especially useful once you need more than two categories, for example, classifying returns as "strong positive," "mild positive," "mild negative," and "strong negative" all in a single call.
  • .shift(1) moves every value in the column down by one row, effectively creating a lagged version of that variable, "yesterday's return," aligned against today's row. Lagged variables are essential in financial modelling whenever you want to ask questions like "does yesterday's return help predict today's?"
  • .rolling(window=5).std() calculates the standard deviation across a moving 5-row window, in this context, a simple rolling measure of short-term volatility. The first four rows will be NaN, since a full 5-day window isn't yet available for them, this is expected and normal behaviour, not a sign of an error.

Why this matters in finance: notice how each new column builds on the one before it: prices become returns, returns become indicators and lags, and lags become rolling statistics. This layered approach, deriving increasingly refined variables from a single original price series, is the everyday backbone of financial and economic data analysis, and it's a pattern you'll see repeated constantly as datasets grow more complex.