Dictionaries

Click here to run this notebook on Colab or click here to download it.

In the previous chapter we used a for loop to read a file and count the words. In this chapter, you’ll learn about a new type called a dictionary, and we’ll use it to count the number of unique words and the number of times each one appears.

Along the way, you will also see how to use an index to select an element from a sequence (tuple, list, or array). And you will learn a little about Unicode, which is used to represent letters, numbers, and punctuation for almost every language in the world.

Indexing

Suppose you have a variable named t that refers to a list or tuple. You can select an element using the bracket operator, []. For example, here’s a tuple of strings:

t = 'zero', 'one', 'two'

To select the first element, we put 0 in brackets:

t[0]
'zero'

To select the second element, we put 1 in brackets:

t[1]
'one'

To select the third element, we put 2 in brackets:

t[2]
'two'

The number in brackets is called an index because it indicates which element we want. Tuples and lists use zero-based numbering; that is, the index of the first element is 0. Some other programming languages use one-based numbering. There are pros and cons of both systems (see https://en.wikipedia.org/wiki/Zero-based_numbering).

The index in brackets can also be a variable:

i = 1
t[i]
'one'

Or an expression with variables, values, and operators:

t[i+1]
'two'

But if the index goes past the end of the list or tuple, you get an error.

Also, the index has to be an integer; if it is any other type, you get an error.

Exercise: You can use negative integers as indices. Try using -1 and -2 as indices, and see if you can figure out what they do.

Dictionaries

A dictionary is similar to a tuple or list, but in a dictionary, the index can be almost any type, not just an integer. We can create an empty dictionary like this:

d = {}

Then we can add elements like this:

d['one'] = 1
d['two'] = 2

In this example, the indices are the strings, 'one' and 'two' If you display the dictionary, it shows each index and the corresponding value.

d
{'one': 1, 'two': 2}

Instead of creating an empty dictionary and then adding elements, you can create a dictionary and specify the elements at the same time:

d = {'one': 1, 'two': 2, 'three': 3}
d
{'one': 1, 'two': 2, 'three': 3}

When we are talking about dictionaries, an index is usually called a key. In this example, the keys are strings and the corresponding values are integers.

A dictionary is also called a map, because it represents correspondence or “mapping”, between keys and values. So we might say that this dictionary maps from English number names to the corresponding integers.

You can use the bracket operator to select an element from a dictionary, like this:

d['two']
2

But don’t forget the quotation marks. Without them, Python looks for a variable named two and doesn’t find one.

To check whether a particular key is in a dictionary, you can use the special word in:

'one' in d
True
'zero' in d
False

The word in is actually an operator in Python, so you can’t use it as a variable name.

If a key is already in a dictionary, adding it again has no effect:

d
{'one': 1, 'two': 2, 'three': 3}
d['one'] = 1
d
{'one': 1, 'two': 2, 'three': 3}

But you can change the value associated with a key:

d['one'] = 100
d
{'one': 100, 'two': 2, 'three': 3}

You can loop through the keys in a dictionary like this:

for key in d:
    print(key)
one
two
three

If you want the keys and the values, one way to get them is to loop through the keys and look up the values:

for key in d:
    print(key, d[key])
one 100
two 2
three 3

Or you can loop through both at the same time, like this:

for key, value in d.items():
    print(key, value)
one 100
two 2
three 3

The items method loops through the key-value pairs in the dictionary; each time through the loop, they are assigned to key and value.

Exercise: Make a dictionary with the numbers 1, 2, and 3 as keys and strings as values. The strings should be the numbers “one”, “two”, and “three” in any language you know.

Write a loop that prints just the values from the dictionary.

Unique words

In the previous chapter we downloaded War and Peace from Project Gutenberg and counted the number of lines and words. Now that we have dictionaries, we can also count the number of unique words and the number of times each one appears.

As we did in the previous chapter, we can read the text of War and Peace and count the number of words.

fp = open('2600-0.txt')
count = 0
for line in fp:
    count += len(line.split())
    
count
566317

To count the number of unique words, I’ll loop through the words in each line and add them as keys in a dictionary:

fp = open('2600-0.txt')
unique_words = {}
for line in fp:
    for word in line.split():
        unique_words[word] = 1

This is the first example we’ve seen with one loop inside another.

  • The outer loop runs through the lines in the file.

  • The inner loops runs through the words in each line.

Each time through the inner loop, we add a word as a key in the dictionary, with the value 1. If the same word appears more than once, it gets added to the dictionary again, which has no effect. So the dictionary contains only one copy of each unique word in the file.

At the end of the loop, we can display the first 10 keys:

i = 0
for key in unique_words:
    print(key)
    i += 1
    if i == 10:
        break
The
Project
Gutenberg
EBook
of
War
and
Peace,
by

The dictionary contains all the words in the file, in order of first appearance. But each word only appears once, so the number of keys is the number of unique words:

len(unique_words)
41991

It looks like there are about 42,000 different words in the book, which is substantially less than the total number of words, about 560,000. But that’s not quite right, because we have not taken into account capitalization and punctuation.

Exercise: Before we deal with that problem, let’s practice with nested loops, that is, one loop inside another. Suppose you have a list of words, like this:

line = ['War', 'and', 'Peace']

Write a nested loop that iterates through each word in the list, and each letter in each word, and prints the letters on separate lines.

Capitalization

When we count unique words, we probably want to treat The and the as the same word. We can do that by converting all words to lower case, using the lower function:

word = 'The'
word.lower()
'the'

lower creates a new string; it does not modify the original string.

word
'The'

However, you can assign the new string back to the existing variable, like this:

word = word.lower()

Now if we can display the new value of word, we get the lowercase version:

word
'the'

Exercise: Modify the previous loop so it makes a lowercase version of each word before adding it to the dictionary. How many unique words are there, if we ignore the difference between uppercase and lowercase?

Punctuation

To remove punctuation from the words, we can use strip, which removes specified characters from the beginning and end of a string. Here’s an example:

word = 'abracadabra'
word.strip('ab')
'racadabr'

In this example, strip removes all instances of a and b from the beginning and end of the word, but not from the middle. But note that it makes a new word; it doesn’t modify the original:

word
'abracadabra'

To remove punctuation, we can use the string library, which provides a variable named punctuation.

import string

string.punctuation
'!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'

string.punctuation contains the most common punctuation marks, but as we’ll see, not all of them. Nevertheless, we can use it to handle most cases. Here’s an example:

line = "It's not given to people to judge what's right or wrong."

for word in line.split():
    word = word.strip(string.punctuation)
    print(word)
It's
not
given
to
people
to
judge
what's
right
or
wrong

strip removes the period at the end of wrong, but not the apostrophes in It's, don't and what's. So that’s good, but we have one more problem to solve. Here’s another line from the book.

line = 'anyone, and so you don’t deserve to have them.”'

Here’s what happens when we try to remove the punctuation.

for word in line.split():
    word = word.strip(string.punctuation)
    print(word)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-2-28f1a7003571> in <module>
      1 for word in line.split():
----> 2     word = word.strip(string.punctuation)
      3     print(word)

NameError: name 'string' is not defined

It removes the comma after anyone, but not the period and quotation mark after them. The problem is that this kind of quotation mark is not in string.punctuation.

To fix this problem, we’ll use the following loop, which

  1. Reads the file and builds a dictionary that contains all punctuation marks that appear in the book, then

  2. It uses the join function to concatenate the keys of the dictionary in a single string.

You don’t have to understand everything about how it works, but you should read it and see how much you can figure out. You can read the documentation of the unicodedata library here at https://docs.python.org/3/library/unicodedata.html.

import unicodedata

fp = open('2600-0.txt')
punc_marks = {}
for line in fp:
    for x in line:
        category = unicodedata.category(x)
        if category[0] == 'P':
            punc_marks[x] = 1
        
all_punctuation = ''.join(punc_marks)
print(all_punctuation)
,.-:[#]*/“’—‘!?”;()%@

Exercise: Modify the word-counting loop from the previous section to convert words to lower case and strip punctuation before adding them to the dictionary. Now how many unique words are there?

Optional: You might want to skip over the front matter and start with the text of Chapter 1, and skip over the license at the end, as we did in the previous chapter.

Word frequencies

In the previous section we counted the number of unique words, but we might also want to know how often each word appears. Then we can find the most common and least common words in the book. To count the frequency of each word, we’ll make a dictionary that maps from each word to the number of times it appears.

Here’s an example that loops through a string and counts the number of times each letter appears.

word = 'Mississippi'

letter_counts = {}
for x in word:
    if x in letter_counts:
        letter_counts[x] += 1
    else:
        letter_counts[x] = 1
        
letter_counts
{'M': 1, 'i': 4, 's': 4, 'p': 2}

The if statement here uses a feature we have not seen before, an else clause. Here’s how it works.

  1. First, it checks whether the letter, x, is already a key in the dictionary, letter_counts.

  2. If so, it runs the first statement, letter_counts[x] += 1, which increments the value associated with the letter.

  3. Otherwise, it runs the second statement, letter_counts[x] = 1, which adds x as a new key, with the value 1 indicating that we have seen the new letter once.

The result is a dictionary that maps from each letter to the number of times it appears.

To get the most common letters, we can use a Counter, which is similar to a dictionary. To use it, we have to import a library called collections:

import collections

Then we use collections.Counter to convert the dictionary to a Counter:

counter = collections.Counter(letter_counts)
type(counter)
collections.Counter

Counter provides a function called most_common we can use to get the most common characters:

counter.most_common(3)
[('i', 4), ('s', 4), ('p', 2)]

The result is a list of tuples, where each tuple contains a character and an integer.

Exercise: Modify the loop from the previous exercise to count the frequency of the words in War and Peace; then print the 20 most common words and the number of times each one appears.

Exercise: You can run most_common with no value in parentheses, like this:

word_freq_pairs = counter.most_common()

The result is a list of tuples, with one tuple for every unique word in the book. Assign the result to a variable so it doesn’t get displayed. Then answer the following questions:

  1. How many times does the #1 ranked word appear (that is, the first element of the list)?

  2. How many times does the #10 ranked word appear?

  3. How many times does the #100 ranked word appear?

  4. How many times does the #1000 ranked word appear?

  5. How many times does the #10000 ranked word appear?

Do you see a pattern in the results? We will explore this pattern more in the next chapter.

Exercise: Write a loop that counts how many words appear 200 times. What are they? How many words appear 100 times, 50 times, and 20 times?

Optional: If you know how to define a function, write a function that takes a Counter and a frequency as arguments, prints all words with that frequency, and returns the number of words with that frequency.