Dictionaries

5. Dictionaries#

Click here to run this notebook on Colab or click here to download it.

In the previous chapter we used a for loop to read a file and count the words. In this chapter we’ll count the number of unique words and the number of times each one appears. To do that, we’ll use one of Python’s most useful features, a dictionary.

You will also see how to select an element from a sequence (tuple, list, or array). And you will learn a little about Unicode, which is used to represent letters, numbers, and punctuation for almost every language in the world.

5.1. Indexing#

Suppose you have a variable named t that refers to a list or tuple. You can select an element using the bracket operator, []. For example, here’s a tuple of strings:

t = ('zero', 'one', 'two')

To select the first element, we put 0 in brackets:

t[0]

'zero'

To select the second element, we put 1 in brackets:

t[1]

'one'

To select the third element, we put 2 in brackets:

t[2]

'two'

The number in brackets is called an index because it indicates which element we want. Tuples and lists use zero-based numbering – that is, the index of the first element is 0. Some other programming languages use one-based numbering.

The index in brackets can also be a variable:

i = 1
t[i]

'one'

Or an expression with variables, values, and operators:

t[i+1]

'two'

But if the index goes past the end of the sequence, you get an error.

%%expect IndexError

t[3]

IndexError: tuple index out of range

Also, the index has to be an integer – if it is any other type, you get an error.

%%expect TypeError

t[1.5]

TypeError: tuple indices must be integers or slices, not float

%%expect TypeError

t['1']

TypeError: tuple indices must be integers or slices, not str

Exercise: You can use negative integers as indices. Try using -1 and -2 as indices, and see if you can figure out what they do.

5.2. Dictionaries#

A dictionary is similar to a tuple or list, but in a dictionary, the index can be almost any type, not just an integer. We can create an empty dictionary like this:

d = {}

Then we can add elements like this:

d['one'] = 1
d['two'] = 2

In this example, the indices are the strings, 'one' and 'two'. If you display the dictionary, it shows each index and the corresponding value.

{'one': 1, 'two': 2}

Instead of creating an empty dictionary and then adding elements, you can create a dictionary and specify the elements at the same time:

d = {'one': 1, 'two': 2, 'three': 3}
d

{'one': 1, 'two': 2, 'three': 3}

When we are talking about dictionaries, an index is usually called a key. In this example, the keys are strings and the corresponding values are integers. A dictionary is also called a map, because it represents a correspondence or “mapping”, between keys and values. So we might say that this dictionary maps from English number names to the corresponding integers.

You can use the bracket operator to select an element from a dictionary, like this:

d['two']

But don’t forget the quotation marks. Without them, Python looks for a variable named two and doesn’t find one.

%%expect NameError

d[two]

NameError: name 'two' is not defined

To check whether a particular key is in a dictionary, you can use the in operator:

'one' in d

True

'zero' in d

False

Because the word in is an operator in Python, you can’t use it as a variable name.

%%expect SyntaxError

in = 5

  Cell In[21], line 1
    in = 5
    ^
SyntaxError: invalid syntax

Each key in a dictionary can only appear once. Adding the same key again has no effect:

d['one'] = 1
d

{'one': 1, 'two': 2, 'three': 3}

But you can change the value associated with a key:

d['one'] = 100
d

{'one': 100, 'two': 2, 'three': 3}

You can loop through the keys in a dictionary like this:

for key in d:
    print(key)

one
two
three

If you want the keys and the values, one way to get them is to loop through the keys and look up the values:

for key in d:
    print(key, d[key])

one 100
two 2
three 3

Or you can loop through both at the same time, like this:

for key, value in d.items():
    print(key, value)

one 100
two 2
three 3

The items method loops through the key-value pairs in the dictionary. Each time through the loop, they are assigned to key and value.

Exercise: Make a dictionary with the integers 1, 2, and 3 as keys and strings as values. The strings should be the words “one”, “two”, and “three” or their equivalents in any language you know.

Write a loop that prints just the values from the dictionary.

5.3. Counting Unique Words#

In the previous chapter we downloaded War and Peace from Project Gutenberg and counted the number of lines and words. Now that we have dictionaries, we can also count the number of unique words and the number of times each one appears.

As we did in the previous chapter, we can read the text of War and Peace and count the number of words.

fp = open('2600-0.txt')
count = 0
for line in fp:
    count += len(line.split())
    
count

To count the number of unique words, we’ll loop through the words in each line and add them as keys in a dictionary:

fp = open('2600-0.txt')
unique_words = {}
for line in fp:
    for word in line.split():
        unique_words[word] = 1

This is the first example we’ve seen with one loop nested inside another.

The outer loop runs through the lines in the file.
The inner loops runs through the words in each line.

Each time through the inner loop, we add a word as a key in the dictionary, with the value 1. If a word that is already in the dictionary appears again, adding it to the dictionary again has no effect. So the dictionary gets only one copy of each unique word in the file. At the end of the loop, we can display the first eight keys like this.

list(unique_words)[:8]

['The', 'Project', 'Gutenberg', 'EBook', 'of', 'War', 'and', 'Peace,']

The list function puts the keys from the dictionary in a list. In the bracket operator, :8 is a special index called a slice that selects the first eight elements.

Each word only appears once, so the number of keys is the number of unique words.

len(unique_words)

There are about 42,000 different words in the book, which is substantially less than the total number of words, about 560,000. But this count is not correct yet, because we have not taken into account capitalization and punctuation.

Exercise: Before we deal with those problems, let’s practice with nested loops – that is, one loop inside another. Suppose you have a list of words, like this:

line = ['War', 'and', 'Peace']

Write a nested loop that iterates through each word in the list, and each letter in each word, and prints the letters on separate lines.

5.4. Dealing with Capitalization#

When we count unique words, we probably want to treat The and the as the same word. We can do that by converting all words to lower case, using the lower function:

word = 'The'
word.lower()

'the'

lower creates a new string; it does not modify the original string.

word

'The'

However, you can assign the new string back to the existing variable, like this:

word = word.lower()

Now if we can display the new value of word, we get the lowercase version:

word

'the'

Exercise: Modify the previous loop so it makes a lowercase version of each word before adding it to the dictionary. How many unique words are there, if we ignore the difference between uppercase and lowercase?

5.5. Removing Punctuation#

To remove punctuation from the words, we can use strip, which removes characters from the beginning and end of a string. Here’s an example:

word = 'abracadabra'
word.strip('ab')

'racadabr'

In this example, strip removes all instances of a and b from the beginning and end of the word, but not from the middle. Like lower, this function makes a new word – it doesn’t modify the original:

word

'abracadabra'

To remove punctuation, we can use the string library, which provides a variable named punctuation.

import string

string.punctuation

'!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'

string.punctuation contains the most common punctuation marks, but as we’ll see, not all of them. Nevertheless, we can use it to handle most cases. Here’s an example:

line = "It's not given to people to judge what's right or wrong."

for word in line.split():
    word = word.strip(string.punctuation)
    print(word)

It's
not
given
to
people
to
judge
what's
right
or
wrong

strip removes the period at the end of wrong, but not the apostrophes in It's, don't and what's. That’s good, because we want to treat an apostrophe as part of a word. But we have one more problem to solve. Here’s another line from the book.

line = 'anyone, and so you don’t deserve to have them.”'

Here’s what happens when we try to remove the punctuation.

for word in line.split():
    word = word.strip(string.punctuation)
    print(word)

anyone
and
so
you
don’t
deserve
to
have
them.”

The comma after anyone is removed, but not the quotation mark at the end of the last word. The problem is that this kind of quotation mark is not in string.punctuation, so strip doesn’t remove it. To fix this problem, we’ll use the following loop, which

Reads the file and builds a dictionary that contains all punctuation marks that appear in the book, then
It uses the join function to concatenate the keys of the dictionary in a single string.

You don’t have to understand everything about how it works, but I suggest you read it and see how much you can figure out.

import unicodedata

fp = open('2600-0.txt')
punc_marks = {}
for line in fp:
    for x in line:
        category = unicodedata.category(x)
        if category[0] == 'P':
            punc_marks[x] = 1
        
all_punctuation = ''.join(punc_marks)
print(all_punctuation)

,.-:[#]*/“’—‘!?”;()%@

The result is a string containing all of the punctuation characters that appear in the document, in the order they first appear.

Exercise: Modify the word-counting loop from the previous section to convert words to lower case and strip punctuation before adding them to the dictionary. Now how many unique words are there?

To get you started, here’s the code from the previous chapter that skips over the front matter at the beginning and the license at the end, as we did in the previous chapter.

fp = open('2600-0.txt')
for line in fp:
    if line.startswith('***'):
        break

count = 0
for line in fp:
    if line.startswith('***'):
        break
    count += len(line.split())

5.6. Counting Word Frequencies#

In the previous section we counted the number of unique words, but we might also want to know how often each word appears. Then we can find the most common and least common words in the book. To count the frequency of each word, we’ll make a dictionary that maps from each word to the number of times it appears.

Here’s an example that loops through a string and counts the number of times each letter appears.

word = 'Mississippi'

letter_counts = {}
for x in word:
    if x in letter_counts:
        letter_counts[x] += 1
    else:
        letter_counts[x] = 1
        
letter_counts

{'M': 1, 'i': 4, 's': 4, 'p': 2}

The if statement includes a feature we have not seen before, an else clause. Here’s how it works.

First, it checks whether the letter, x, is already a key in the dictionary, letter_counts.
If so, it runs the first statement, letter_counts[x] += 1, which increments the value associated with the letter.
Otherwise, it runs the second statement, letter_counts[x] = 1, which adds x as a new key, with the value 1 indicating that we have seen the new letter once.

The result is a dictionary that maps from each letter to the number of times it appears. To get the most common letters, we can use a Counter, which is similar to a dictionary. To use it, we have to import a library called collections:

import collections

Then we use collections.Counter to convert the dictionary to a Counter:

counter = collections.Counter(letter_counts)
type(counter)

collections.Counter

Counter provides a function called most_common we can use to get the most common characters:

counter.most_common(3)

[('i', 4), ('s', 4), ('p', 2)]

The result is a list of tuples, where each tuple contains a character and count, sorted by count.

Exercise: Modify the loop from the previous exercise to count the frequency of the words in War and Peace. Then print the 20 most common words and the number of times each one appears.

Exercise: You can run most_common with no value in parentheses, like this:

word_freq_pairs = counter.most_common()

The result is a list of tuples, with one tuple for every unique word in the book. Use it to answer the following questions:

How many times does the #1 ranked word appear (that is, the first element of the list)?
How many times does the #10 ranked word appear?
How many times does the #100 ranked word appear?
How many times does the #1000 ranked word appear?
How many times does the #10000 ranked word appear?

Do you see a pattern in the results? We will explore this pattern more in the next chapter.

Exercise: Write a loop that counts how many words appear 200 times. What are they? How many words appear 100 times, 50 times, and 20 times?

Optional: If you know how to define a function, write a function that takes a Counter and a frequency as arguments, prints all words with that frequency, and returns the number of words with that frequency.

5.7. Summary#

This chapter introduces dictionaries, which are collections of keys and corresponding values. We used a dictionary to count the number of unique words in a file and the number of times each one appears.

It also introduces the bracket operator, which selects an element from a list or tuple, or looks up a key in a dictionary and finds the corresponding value.

We saw some new methods for working with strings, including lower and strip. Also, we used the unicodedata library to identify characters that are considered punctuation.