Loops and Files#

Click here to run this notebook on Colab or click here to download it.

This chapter presents loops, which are used to represent repeated computation, and files, which are used to store data. As an example, we will download the famous book War and Peace from Project Gutenberg and write a loop that reads the book and counts the words. This example presents some new computational tools; it is also an introduction to working with textual data.

Loops#

One of the most important elements of computation is repetition, and the most common way to represent repetition is a for loop. As a simple example, suppose we want to display the elements of a tuple. Here’s a tuple of three integers:

t = 1, 2, 3

And here’s a for loop that prints the elements.

for x in t:
    print(x)

1
2
3

The first line of the loop is a header that specifies the tuple, t, and a variable name, x. The tuple already exists, but x does not; the loop will create it. Note that the header ends with a colon, :.

Inside the loop is a print statement, which displays the value of x.

So here’s what happens:

When the loop starts, it gets the first element of t, which is 1, and assigns it to x. It executes the print statement, which displays the value 1.
Then it gets the second element of t, which is 2, and displays it.
Then it gets the third element of t, which is 3, and displays it.

After printing the last element of the tuple, the loop ends.

We can also loop through the letters in a string:

word = 'Data'

for letter in word:
    print(letter)

D
a
t
a

When the loop begins, word already exists, but letter does not. Again, the loop creates letter and assigns values to it.

The variable created by the loop is called the loop variable. You can give it any name you like; in this example, I chose letter to remind me what kind of value it contains.

After the loop ends, the loop variable contains the last value.

letter

'a'

Exercise: Create a list, called sequence with four elements of any type. Write a for loop that prints the elements. Call the loop variable element.

You might wonder why I didn’t call the list list. I avoided it because Python has a function named list that makes new lists. For example, if you have a string, you can make a list of letters, like this:

list('string')

['s', 't', 'r', 'i', 'n', 'g']

If you create a variable named list, you can’t use the function any more.

Counting with Loops#

War and Peace is a famously long book; let’s see how long it is. To count the words we need two elements: looping through the words in a text, and counting. We’ll start with counting.

We’ve already seen that you can create a variable and give it a value, like this:

count = 0
count

If you assign a different value to the same variable, the new value replaces the old one.

count = 1
count

You can increase the value of a variable by reading the old value, adding 1, and assigning the result back to the original variable.

count = count + 1
count

Increasing the value of a variable is called incrementing; decreasing the value is called decrementing. These operations are so common that there are special operators for them.

count += 1
count

In this example, the += operator reads the value of count, adds 1, and assigns the result back to count. Python also provides -= and other update operators like *= and /=.

Exercise: The following is a number trick from Learn With Math Games at https://www.learn-with-math-games.com/math-number-tricks.html:

Finding Someone’s Age

Ask the person to multiply the first number of their age by 5.

Tell them to add 3.

Now tell them to double this figure.

Finally, have the person add the second number of their age to the figure and have them tell you the answer.

Deduct 6 and you will have their age.

Test this algorithm using your age. Use a single variable and update it using += and other update operators.

Files#

Now that we know how to count, let’s see how we can read words from a file. We can download War and Peace from Project Gutenberg, which is a repository of free books at https://www.gutenberg.org.

In order to read the contents of the file, you have to open it, which you can do with the open function.

fp = open('2600-0.txt')
fp

<_io.TextIOWrapper name='2600-0.txt' mode='r' encoding='UTF-8'>

The result is a TextIOWrapper, which is a type of file pointer. It contains the name of the file, the mode (which is r for “reading”) and the encoding (which is UTF for “Unicode Transformation Format”). A file pointer is like a bookmark; it keeps track of which parts of the file you have read.

If you use a file pointer in a for loop, it loops through the lines in the file. So we can count the number of lines like this:

fp = open('2600-0.txt')
count = 0
for line in fp:
    count += 1

And then display the result.

count

There are about 66,000 lines in this file.

if Statements#

We’ve already see comparison operators, like > and <, which compare values and produce a Boolean result, True or False. For example, we can compare the final value of count to a number:

count > 60000

True

We can use a comparison operator in an if statement to check for a condition and take action accordingly.

if count > 60000:
    print('Long book!')

Long book!

The first line of the if statement specifies the condition we’re checking for. Like the header of a for statement, the first line of an if statement has to end with a colon.

If the condition is true, the indented statement runs; otherwise, it doesn’t. In the previous example, the condition is true, so the print statement runs. In the following example, the condition is false, so the print statement doesn’t run.

if count < 1000:
    print('Short book!')

We can put a print statement inside a for loop. In this example, we only print a line from the book when count is 1. The other lines are read, but not displayed.

fp = open('2600-0.txt')
count = 0
for line in fp:
    if count == 1:
        print(line)
    count += 1

The Project Gutenberg EBook of War and Peace, by Leo Tolstoy

Notice the indentation in this example:

Statements inside the for loop are indented.
The statement inside the if statement is indented.
The statement count += 1 is outdented from the previous line, so it ends the if statement. But it is still inside the for loop.

It is legal in Python to use spaces or tabs for indentation, but the most common convention is to use four spaces, never tabs. That’s what I’ll do in my code and I strongly suggest you follow the convention.

The break Statement#

If we display the final value of count, we see that the loop reads the entire file, but only prints one line:

count

We can avoid reading the whole file by using a break statement, like this:

fp = open('2600-0.txt')
count = 0
for line in fp:
    if count == 1:
        print(line)
        break
    count += 1

The Project Gutenberg EBook of War and Peace, by Leo Tolstoy

The break statement ends the loop immediately, skipping the rest of the file. We can confirm that by checking the last value of count:

count

Exercise: Write a loop that prints the first 5 lines of the file and then breaks out of the loop.

Whitespace#

If we run the loop again and display the final value of line, we see the special sequence \n at the end.

fp = open('2600-0.txt')
count = 0
for line in fp:
    if count == 1:
        break
    count += 1

line

'The Project Gutenberg EBook of War and Peace, by Leo Tolstoy\n'

This sequence represents a single character, called a newline, that puts vertical space between lines. If we use a print statement to display line, we don’t see the special sequence, but we do see extra space after the line.

print(line)

The Project Gutenberg EBook of War and Peace, by Leo Tolstoy

In other strings, you might see the sequence \t, which represents a “tab” character. When you print a tab character, it adds enough space to make the next character appear in a column that is a multiple of 8.

print('01234567' * 6)
print('a\tbc\tdef\tghij\tklmno\tpqrstu')

012345670123456701234567012345670123456701234567
a	bc	def	ghij	klmno	pqrstu

Newline characters, tabs, and spaces are called whitespace because when they are printed they leave white space on the page (assuming that the background color is white).

Counting Words#

So far we’ve managed to count the lines in a file, but each line contains several words. To split a line into words, we can use a function called split that returns a list of words. To be more precise, split doesn’t actually know what a word is; it just splits the line wherever there’s a space or other whitespace character.

line.split()

['The',
 'Project',
 'Gutenberg',
 'EBook',
 'of',
 'War',
 'and',
 'Peace,',
 'by',
 'Leo',
 'Tolstoy']

Notice that the syntax for split is different from other functions we have seen. Normally when we call a function, we name the function and provide values in parentheses. So you might have expected to write split(line). Sadly, that doesn’t work.

The problem is that the split function belongs to the string line; in a sense, the function is attached to the string, so we can only refer to it using the string and the dot operator (the period between line and split). For historical reasons, functions like this are called methods.

Now that we can split a line into a list of words, we can use len to get the number of words in each list, and increment count accordingly.

fp = open('2600-0.txt')
count = 0
for line in fp:
    count += len(line.split())

count

By this count, there are more than half a million words in War and Peace.

Actually, there aren’t quite that many, because the file we got from Project Gutenberg has some introductory text and a table of contents before the text. And it has some license information at the end. To skip this “front matter”, we can use one loop to read lines until we get to CHAPTER I, and then a second loop to count the words in the remaining lines.

The file pointer, fp, keeps track of where it is in the file, so the second loop picks up where the first loop leaves off. In the second loop, we check for the end of the book and stop, so we ignore the “back matter” at the end of the file.

first_line = "CHAPTER I\n"
last_line = "End of the Project Gutenberg EBook of War and Peace, by Leo Tolstoy\n"

fp = open('2600-0.txt')
for line in fp:
    if line == first_line:
        break

count = 0
for line in fp:
    if line == last_line:
        print(line)
        break
    count += len(line.split())

End of the Project Gutenberg EBook of War and Peace, by Leo Tolstoy

count

Two things to notice about this program:

When we compare two values to see if they are equal, we use the == operator, not to be confused with =, which is the assignment operator.
The string we compare line to has a newline at the end. If we leave that out, it doesn’t work.

Exercise:

In the previous program, replace == with = and see what happens. This is a common error, so it is good to see what the error message looks like.
Correct the previous error, then remove the newline character after CHAPTER I, and see what happens.

The first error is a syntax error, which means that the program violates the rules of Python. If your program has a syntax error, the Python interpreter prints an error message, and the program never runs.

The second error is a logic error, which means that there is something wrong with the logic of the program. The syntax is legal, and the program runs, but it doesn’t do what we wanted. Logic errors can be hard to find because we don’t get any error messages.

If you have a logic error, here are two strategies for debugging:

Add print statements so the program displays additional information while it runs.
Simplify the program until it does what you expect, and then gradually add more code, testing as you go.

Summary#

This chapter presents loops, if statements, and the break statement. It also introduces tools for working with letters and words, and a simple kind of textual analysis, word counting.

In the next chapter we’ll continue this example, counting the number of unique words in a text and the number of times each word appears. And we’ll see one more way to represent a collection of values, a Python dictionary.

Elements of Data Science

Loops and Files

Contents