Text analysis with Python#
Copyright 2019–2026 Allen Downey
Click here to run this notebook on Colab
from os.path import basename, exists
def download(url):
filename = basename(url)
if not exists(filename):
from urllib.request import urlretrieve
local, _ = urlretrieve(url, filename)
print("Downloaded " + local)
BASE = "https://github.com/AllenDowney/CompStats/raw/master/"
download(BASE + "pg42671.txt")
%matplotlib inline
import matplotlib.pyplot as plt
Word Frequencies#
Let’s look at frequencies of words, bigrams and trigrams in a text.
The following function reads lines from a file or URL and splits them into words:
def iterate_words(filename):
"""Read lines from a file and split them into words."""
for line in open(filename):
for word in line.split():
yield word.strip()
Here’s an example using a book. wc is a Counter of words, that is, a dictionary that maps from each word to the number of times it appears:
# PRIDE AND PREJUDICE
# By Jane Austen
# https://www.gutenberg.org/cache/epub/42671/pg42671.txt
filename = 'pg42671.txt'
from collections import Counter
wc = Counter(iterate_words(filename))
Here are the 20 most common words:
wc.most_common(20)
[('the', 4230),
('to', 4117),
('of', 3689),
('and', 3350),
('a', 1930),
('her', 1831),
('in', 1821),
('was', 1797),
('I', 1696),
('that', 1432),
('not', 1368),
('she', 1303),
('be', 1203),
('his', 1168),
('as', 1128),
('had', 1126),
('with', 1044),
('he', 1038),
('for', 1018),
('you', 980)]
Word frequencies in natural languages follow a predictable pattern called Zipf’s law (which is an instance of Stigler’s law, which is also an instance of Stigler’s law).
We can see the pattern by lining up the words in descending order of frequency and plotting their counts (4230, 4117, 3689) versus ranks (1st, 2nd, 3rd, …):
def counter_ranks(wc):
"""Returns ranks and counts as lists."""
return zip(*enumerate(sorted(wc.values(), reverse=True)))
ranks, counts = counter_ranks(wc)
plt.plot(ranks, counts)
plt.xlabel('Rank')
plt.ylabel('Count')
plt.title('Word count versus rank, linear scale');
Huh. Maybe that’s not so clear after all. The problem is that the counts drop off very quickly. If we use the highest count to scale the figure, most of the other counts are indistinguishable from zero.
Also, there are more than 10,000 words, but most of them appear only a few times, so we are wasting most of the space in the figure in a regime where nothing is happening.
This kind of thing happens a lot. A common way to deal with it is to compute the log of the quantities or to plot them on a log scale:
ranks, counts = counter_ranks(wc)
plt.plot(ranks, counts)
plt.xlabel('Rank')
plt.ylabel('Count')
plt.xscale('log')
plt.yscale('log')
plt.title('Word count versus rank, log-log scale');
This (approximately) straight line is characteristic of Zipf’s law.
n-grams#
On to the next topic: bigrams and trigrams.
from itertools import tee
def pairwise(iterator):
"""Iterates through a sequence in overlapping pairs.
If the sequence is 1, 2, 3, the result is (1, 2), (2, 3), (3, 4), etc.
"""
a, b = tee(iterator)
next(b, None)
return zip(a, b)
bigrams is the histogram of word pairs:
bigrams = Counter(pairwise(iterate_words(filename)))
And here are the 20 most common:
bigrams.most_common(20)
[(('of', 'the'), 488),
(('to', 'be'), 421),
(('in', 'the'), 389),
(('to', 'the'), 261),
(('of', 'her'), 246),
(('I', 'am'), 233),
(('of', 'his'), 225),
(('had', 'been'), 194),
(('could', 'not'), 167),
(('that', 'he'), 165),
(('she', 'had'), 164),
(('for', 'the'), 162),
(('and', 'the'), 160),
(('it', 'was'), 156),
(('she', 'was'), 147),
(('he', 'had'), 143),
(('on', 'the'), 143),
(('I', 'have'), 140),
(('to', 'her'), 139),
(('such', 'a'), 137)]
Similarly, we can iterate the trigrams:
def triplewise(iterator):
a, b, c = tee(iterator, 3)
next(b)
next(c)
next(c)
return zip(a, b, c)
And make a Counter:
trigrams = Counter(triplewise(iterate_words(filename)))
Here are the 20 most common:
trigrams.most_common(20)
[(('as', 'soon', 'as'), 47),
(('she', 'could', 'not'), 43),
(('I', 'do', 'not'), 38),
(('that', 'he', 'had'), 36),
(('I', 'am', 'sure'), 34),
(('could', 'not', 'be'), 30),
(('as', 'well', 'as'), 28),
(('it', 'would', 'be'), 27),
(('by', 'no', 'means'), 26),
(('would', 'have', 'been'), 26),
(('that', 'he', 'was'), 26),
(('that', 'it', 'was'), 25),
(('that', 'she', 'had'), 25),
(('one', 'of', 'the'), 24),
(('I', 'am', 'not'), 23),
(('and', 'she', 'was'), 23),
(('the', 'Project', 'Gutenberg'), 22),
(('not', 'to', 'be'), 21),
(('it', 'was', 'not'), 21),
(('*', '*', '*'), 21)]
Markov analysis#
And now for a little fun. I’ll make a dictionary that maps from each word pair to a Counter of the words that can follow.
from collections import defaultdict
d = defaultdict(Counter)
for a, b, c in trigrams:
d[a, b][c] += trigrams[a, b, c]
Now we can look up a pair and see what might come next:
d['Mr.', 'Darcy']
Counter({'had': 12,
'was': 9,
'is': 7,
'would': 6,
'has': 4,
'might': 4,
'and': 4,
'with': 3,
'looked': 3,
'could': 3,
'to': 3,
'then': 2,
'took': 2,
'in': 2,
'may': 2,
'it': 2,
'so': 2,
'gave': 2,
'than': 2,
'should': 2,
'soon': 1,
'danced': 1,
'walked': 1,
'speaking': 1,
'mean,"': 1,
'only': 1,
'stood': 1,
'bowed.': 1,
'replied': 1,
'smiled;': 1,
'felt': 1,
'did': 1,
'corroborated': 1,
'just': 1,
'without': 1,
'bequeathed': 1,
'chose': 1,
'liked': 1,
'often': 1,
'is."': 1,
'can': 1,
'contradict': 1,
'he': 1,
'more': 1,
'seemed': 1,
'for': 1,
'before': 1,
'by': 1,
'they': 1,
'spoke': 1,
'only,': 1,
'drew': 1,
'related': 1,
'came': 1,
'give': 1,
'himself': 1,
'walk': 1,
'changed': 1,
'improves': 1,
'approaching': 1,
'invite': 1,
'handed': 1,
'himself;': 1,
'called': 1,
'afterwards': 1,
'were': 1,
'appeared.': 1,
'acquainted': 1,
'called,': 1,
'asked': 1,
'found,': 1,
'looked,': 1,
'acknowledged,': 1,
'returned': 1,
'must': 1,
'rise': 1,
'appeared': 1,
'sends': 1,
'exposed': 1,
'half': 1,
'about': 1})
Here are the most common words that follow “into the”:
d['into', 'the'].most_common(10)
[('room.', 4),
('breakfast', 3),
('neighbourhood."', 2),
('country', 2),
('dining-room,', 2),
('breakfast-parlour,', 1),
('room', 1),
('avenue."', 1),
('drawing-room;', 1),
('drawing-room', 1)]
The following function chooses a random word from the suffixes in a Counter:
import random
def choice(counter):
"""Chooses a random element."""
return random.choice(list(counter.elements()))
choice(d['into', 'the'])
'house--but'
Given a prefix, we can choose a random suffix:
prefix = 'into', 'the'
suffix = choice(d[prefix])
suffix
'breakfast'
Then we can shift the words and compute the next prefix:
prefix = prefix[1], suffix
prefix
('the', 'breakfast')
Repeating this process, we can generate random new text that has the same correlation structure between words as the original:
for i in range(100):
suffix = choice(d[prefix])
print(suffix, end=' ')
prefix = prefix[1], suffix
room, "and what do you really think so?" cried Elizabeth, "that I ever before mentioned to Miss Bingley could not be under any alarm. I will sit in my opinion, exertion should always be properly guarded and attended, according to Charlotte's first sketch. She was at Rosings?" "Nearly three weeks." She could think with wonder, of Mr. Collins, "I am glad of one individual, change the meaning of principle which he had been formed, as he was to come without it. "Yet it is lessening the honour of calling in Grosvenor-street." She wrote even to pass your life." "You make
With a prefix of two words, we typically get text that flirts with sensibility.