Testing a Difference in Slopes#

Here’s a question from the Reddit statistics forum.

How do I determine if the difference in change-speed is significant?

Hi! I’m doing a project for uni and I’m in need of a statistical test but I’m not well read enough on statistics to figure out which.

I have 7 years worth of anual data (average plant diversity, using Shannon Wiener) from two separate locations and I need to figure out if there is a significant difference between the two locations in the speed at which the diversity changes.

Let me know if any additional information is needed to figure this out!

We can do this by resampling, but we’ll need to think about how to model the data-generating process and the null hypothesis. One of the best ways to model the data-generating process is to generate some random data.

Click here to run this notebook on Colab.

I’ll download a utilities module with some of my frequently-used functions, and then import the usual libraries.

Hide code cell content

from os.path import basename, exists

def download(url):
    filename = basename(url)
    if not exists(filename):
        from urllib.request import urlretrieve

        local, _ = urlretrieve(url, filename)
        print("Downloaded " + str(local))
    return filename

download('https://github.com/AllenDowney/DataQnA/raw/main/nb/utils.py');

Hide code cell content

# install the empiricaldist library, if necessary

try:
    import empiricaldist
except ImportError:
    !pip install empiricaldist

Hide code cell content

import numpy as np
np.set_printoptions(legacy='1.25')
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns

from utils import decorate

Answer#

I’ll start with a range of 7 years.

n = 7
xs = np.arange(n)

And generate fake datasets with different intercepts and slopes, but the same standard deviation of the error terms, sigma.

np.random.seed(1)

sigma = 1
ys1 = 10 - 1 * xs + np.random.normal(0, sigma, size=n)
ys2 = 8 - 1.5 * xs + np.random.normal(0, sigma, size=n)

Here’s what the data look like.

plt.plot(xs, ys1, 'o')
plt.plot(xs, ys2, 'o')

decorate(xlabel='x', ylabel='y')
_images/e0be04e5eb33a60e3cfe60e9e5b40c1c8216c891fed89e79c139bda5661d3e11.png

And here are the estimated slopes for the two datasets separately.

from scipy.stats import linregress

res1 = linregress(xs, ys1)
res1.slope
-1.0580209279718773
res2 = linregress(xs, ys2)
res2.slope
-1.570079474838621

Here’s the observed difference in the slopes.

diff_actual = res1.slope - res2.slope
diff_actual
0.5120585468667438

The null hypothesis#

Now, if the question is whether this difference might be due to chance, we have to define what we mean by “due to chance”. In this example, the null hypothesis is that the slope is the same, so the first question we have to answer is, “If the slope is the same, what is it?”

To answer that, we can use multiple regression with both datasets, constrained so they have the same slope, but allowing them to have different intercepts. First I’ll combine the datasets in a single DataFrame, with a column indicating which group each row is from.

df1 = pd.DataFrame(dict(x=xs, y=ys1))
df1['group'] = 1
df1
x y group
0 0 11.624345 1
1 1 8.388244 1
2 2 7.471828 1
3 3 5.927031 1
4 4 6.865408 1
5 5 2.698461 1
6 6 5.744812 1
df2 = pd.DataFrame(dict(x=xs, y=ys2))
df2['group'] = 2
df2
x y group
0 0 7.238793 2
1 1 6.819039 2
2 2 4.750630 2
3 3 4.962108 2
4 4 -0.060141 2
5 5 0.177583 2
6 6 -1.384054 2
df = pd.concat([df1, df2]).reset_index(drop=True)
df
x y group
0 0 11.624345 1
1 1 8.388244 1
2 2 7.471828 1
3 3 5.927031 1
4 4 6.865408 1
5 5 2.698461 1
6 6 5.744812 1
7 0 7.238793 2
8 1 6.819039 2
9 2 4.750630 2
10 3 4.962108 2
11 4 -0.060141 2
12 5 0.177583 2
13 6 -1.384054 2

Now we can run the regression.

import warnings

warnings.filterwarnings('ignore', message='kurtosistest*')
import statsmodels.formula.api as smf

formula = 'y ~ x + C(group)'
result = smf.ols(formula, data=df).fit()
result.summary()
OLS Regression Results
Dep. Variable: y R-squared: 0.860
Model: OLS Adj. R-squared: 0.834
Method: Least Squares F-statistic: 33.73
Date: Fri, 13 Nov 2026 Prob (F-statistic): 2.03e-05
Time: 13:25:44 Log-Likelihood: -23.570
No. Observations: 14 AIC: 53.14
Df Residuals: 11 BIC: 55.06
Df Model: 2
Covariance Type: nonrobust
coef std err t P>|t| [0.025 0.975]
Intercept 10.9022 0.810 13.461 0.000 9.120 12.685
C(group)[T.2] -3.7452 0.786 -4.766 0.001 -5.475 -2.016
x -1.3141 0.196 -6.690 0.000 -1.746 -0.882
Omnibus: 0.732 Durbin-Watson: 2.652
Prob(Omnibus): 0.694 Jarque-Bera (JB): 0.688
Skew: 0.425 Prob(JB): 0.709
Kurtosis: 2.323 Cond. No. 9.27


Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

Under the assumption that the slope is the same in both groups, the estimated slope is about -1.2, and the intercepts for the groups are different.

We can use predict to generate fitted values for each group.

fit1 = result.predict(df1)
fit2 = result.predict(df2)

And here’s what they look like compared to the data.

plt.plot(xs, ys1, 'o')
plt.plot(xs, fit1, color='C0')
plt.plot(xs, ys2, 'o')
plt.plot(xs, fit2, color='C1')

decorate(xlabel='x', ylabel='y')
_images/2491085d4c6a9db92ae3de6235b2a04ae355164058cac1e8e3bc0c945ff515db.png

Simulating the null hypothesis#

To compute a p-value, we have to compute the test statistic – the difference in slopes – for datasets generated under the null hypothesis. We can do that by choosing random residuals and adding them to the fitted lines.

Here’s what the residuals look like.

resid1 = ys1 - fit1
resid2 = ys2 - fit2

plt.plot(xs, resid1, 'o')
plt.plot(xs, resid2, 'o')
decorate(xlabel='x', ylabel='Residual')
_images/523cb64f959baf21403732cde784f5d669551406326d4952f919320441f34b72.png
resid = np.concatenate([resid1, resid2])

The following function generates random datasets under the null hypothesis and computes the difference in their slopes.

def resample_slope():
    resampled1 = fit1 + np.random.choice(resid, size=n, replace=True)
    resampled2 = fit2 + np.random.choice(resid, size=n, replace=True)

    res1 = linregress(xs, resampled1)
    res2 = linregress(xs, resampled2)
    diff = res1.slope - res2.slope
    return diff

If we call this function once, we get a single instance of the test statistic under the null hypothesis – this is the answer to the question, “If there were actually no difference in the slopes and we ran the experiment again, how big a difference might we see by chance?”

np.random.seed(17)
resample_slope()
0.4988292071804228

If we run it many times, we get a sample of the test statistic under the null hypothesis.

np.random.seed(17)
diffs = [resample_slope() for i in range(1001)]

Here’s what the distribution of the test statistic looks like.

sns.kdeplot(diffs)
plt.axvline(diff_actual, color='0.7', ls=':')
decorate(xlabel='Difference in slopes', ylabel='Density')
_images/2dbe27e4582bf4f5847630c7565ae7fb510f4bc5e4641ddc1acd5a515f111c92.png

The mean is near 0, as we expect if there were actually no difference in the slopes. But the tails extend past the observed difference, which is near 0.5. That means that a difference as big as 0.5 could plausibly happen by chance.

To quantify how plausible, we can compute the p-value, which is the fraction of the sample that exceeds the observed difference in magnitude.

p_value = np.mean(np.abs(diffs) >= np.abs(diff_actual))
p_value
0.14085914085914086

For the fake dataset I generated, the p-value is about 14%, which means that the observed difference could plausibly happen by chance, even if the slopes are actually the same in the two groups.

The parametric version#

The resampling method draws residuals from the ones we observed, so it makes no assumptions about their distribution. Alternatively, we can assume that the residuals follow a normal distribution, and draw them from a normal distribution with the same mean and standard deviation as the observed residuals.

m, s = np.mean(resid), np.std(resid)

Here’s a version of resample_slope that uses the normal model.

def resample_slope2():
    resampled1 = fit1 + np.random.normal(m, s, size=n)
    resampled2 = fit2 + np.random.normal(m, s, size=n)

    res1 = linregress(xs, resampled1)
    res2 = linregress(xs, resampled2)
    diff = res1.slope - res2.slope
    return diff

If we run it many times, we get another sample of the test statistic under the null hypothesis.

np.random.seed(18)
diffs2 = [resample_slope2() for i in range(1001)]

Here’s how the two distributions compare.

sns.kdeplot(diffs, label='Resampled residuals')
sns.kdeplot(diffs2, label='Normal residuals')
plt.axvline(diff_actual, color='0.7', ls=':')
decorate(xlabel='Difference in slopes', ylabel='Density')
_images/866642ed65da42a6ba953d668612d1e4210ab18d8d6a6fe951347600dfdc6de2.png

They are similar, so the p-values are similar, too.

p_value2 = np.mean(np.abs(diffs2) >= np.abs(diff_actual))
p_value2
0.14785214785214784

Finally, the same hypothesis can be tested with a regression model that includes an interaction term, which allows the slope to be different in the two groups. The p-value for the interaction term tests whether the difference in slopes is statistically significant.

result_inter = smf.ols('y ~ x * C(group)', data=df).fit()
result_inter.pvalues['x:C(group)[T.2]']
0.20632339688522164

This p-value is bigger, about 21%, because the regression model is more conservative than the simulations. The simulations use the observed residuals, which are a little smaller than the errors in the data-generating process, because the fitted lines are chosen to minimize them. The regression model corrects for that, and it also accounts for the uncertainty in estimating the size of the errors from only 14 data points. With such a small dataset, those corrections make a noticeable difference. But the conclusion is the same: a difference in slopes this big could plausibly be due to chance.

Discussion#

To answer the question: if you have a time series of annual measurements from two locations, you can test whether the rate of change is different by estimating the slope at each location and asking whether the difference could plausibly be due to chance. The simplest way to do that is a regression model with an interaction term, as in the last step. The resampling methods ask the same question, but they make it explicit how we model the data-generating process and the null hypothesis.

With only 7 years of data, though, the test has low power. In the fake data I generated, the slopes really are different – by 0.5 per year – but the difference we see in the data is not statistically significant.

So if the test with real data is not significant, that doesn’t mean the rates of change are the same; it might mean there isn’t enough data to tell. It might be more informative to report the estimated difference in slopes with a confidence interval, which shows how big the difference could be.

Also, annual measurements from the same location might not be independent – a good year might be followed by another good year – which these tests don’t take into account.

Data Q&A: Answering the real questions with Python

Copyright 2024–2026 Allen B. Downey

License: Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International