The Dormouse's story
\n", "\n", "Once upon a time there were three little sisters; and their names were\n", "Elsie,\n", "Lacie and\n", "Tillie;\n", "and they lived at the bottom of a well.
\n", "\n", "...
\n", "\"\"\"" ] }, { "cell_type": "markdown", "id": "88220a73", "metadata": {}, "source": [ "Here's how we use BeautifulSoup to read it." ] }, { "cell_type": "code", "execution_count": 2, "id": "397bda6b", "metadata": { "tags": [ "fill-in" ] }, "outputs": [ { "data": { "text/plain": [ "bs4.BeautifulSoup" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from bs4 import BeautifulSoup\n", "\n", "soup = BeautifulSoup(html_doc)\n", "type(soup)" ] }, { "cell_type": "markdown", "id": "5313d704", "metadata": {}, "source": [ "The result is a `BeautifulSoup` object that represents the root of the tree. If we display the soup, it reproduces the HTML." ] }, { "cell_type": "code", "execution_count": 3, "id": "d9cd7d4a", "metadata": { "tags": [ "fill-in" ] }, "outputs": [ { "data": { "text/plain": [ "The Dormouse's story
\n", "Once upon a time there were three little sisters; and their names were\n", "Elsie,\n", "Lacie and\n", "Tillie;\n", "and they lived at the bottom of a well.
\n", "...
\n", "" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "soup" ] }, { "cell_type": "markdown", "id": "6edc272a", "metadata": {}, "source": [ "`prettify` uses indentation to show the structure of the document." ] }, { "cell_type": "code", "execution_count": 4, "id": "179128f7", "metadata": { "tags": [ "fill-in" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", " \n", "\n", " \n", " The Dormouse's story\n", " \n", "
\n", "\n", " Once upon a time there were three little sisters; and their names were\n", " \n", " Elsie\n", " \n", " ,\n", " \n", " Lacie\n", " \n", " and\n", " \n", " Tillie\n", " \n", " ;\n", "and they lived at the bottom of a well.\n", "
\n", "\n", " ...\n", "
\n", " \n", "\n" ] } ], "source": [ "print(soup.prettify())" ] }, { "cell_type": "markdown", "id": "1bd74af3", "metadata": {}, "source": [ "The `BeautifulSoup` object has a property called `children` that returns an iterator of the objects it contains." ] }, { "cell_type": "code", "execution_count": 5, "id": "0141df22", "metadata": { "tags": [ "fill-in" ] }, "outputs": [ { "data": { "text/plain": [ "The Dormouse's story
\n", "Once upon a time there were three little sisters; and their names were\n", " Elsie,\n", " Lacie and\n", " Tillie;\n", " and they lived at the bottom of a well.
\n", "...
\n", " ]" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "soup.contents" ] }, { "cell_type": "markdown", "id": "26dfbfd3", "metadata": {}, "source": [ "The only child is an HTML element that contains the whole document.\n", "Let's get just this element:" ] }, { "cell_type": "code", "execution_count": 10, "id": "6d83d8f0", "metadata": { "tags": [ "fill-in" ] }, "outputs": [ { "data": { "text/plain": [ "The Dormouse's story
\n", "Once upon a time there were three little sisters; and their names were\n", "Elsie,\n", "Lacie and\n", "Tillie;\n", "and they lived at the bottom of a well.
\n", "...
\n", "" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "element = soup.contents[0]\n", "element" ] }, { "cell_type": "markdown", "id": "68b7b64d", "metadata": {}, "source": [ "The type of the element is `Tag`:" ] }, { "cell_type": "code", "execution_count": 11, "id": "6aae480b", "metadata": { "tags": [ "fill-in" ] }, "outputs": [ { "data": { "text/plain": [ "bs4.element.Tag" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "type(element)" ] }, { "cell_type": "markdown", "id": "2f9b9cad", "metadata": {}, "source": [ "And the name of the tag is `html`." ] }, { "cell_type": "code", "execution_count": 12, "id": "e11a78d8", "metadata": { "tags": [ "fill-in" ] }, "outputs": [ { "data": { "text/plain": [ "'html'" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "element.name" ] }, { "cell_type": "markdown", "id": "d2e65f4e", "metadata": {}, "source": [ "Now let's get the children of this top-level element:" ] }, { "cell_type": "code", "execution_count": 13, "id": "c5a6fd88", "metadata": { "tags": [ "fill-in" ] }, "outputs": [ { "data": { "text/plain": [ "[The Dormouse's story
\n", "Once upon a time there were three little sisters; and their names were\n", " Elsie,\n", " Lacie and\n", " Tillie;\n", " and they lived at the bottom of a well.
\n", "...
\n", " ]" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "children = element.contents\n", "children" ] }, { "cell_type": "markdown", "id": "60cd822d", "metadata": {}, "source": [ "There are three elements in this list, but it's hard to read because when you print an element, it prints all of the HTML.\n", "\n", "I'll use the following function to print elements in a simple form." ] }, { "cell_type": "code", "execution_count": 14, "id": "9860d35c", "metadata": {}, "outputs": [], "source": [ "from bs4 import Tag, NavigableString\n", "\n", "def print_element(element):\n", " if isinstance(element, Tag):\n", " print(f'{type(element).__name__}<{element.name}>')\n", " if isinstance(element, NavigableString):\n", " print(type(element).__name__)" ] }, { "cell_type": "code", "execution_count": 15, "id": "9cc4c70d", "metadata": { "tags": [ "fill-in" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Tag\n" ] } ], "source": [ "print_element(element)" ] }, { "cell_type": "markdown", "id": "e4f38388", "metadata": {}, "source": [ "And the following function to print a list of elements." ] }, { "cell_type": "code", "execution_count": 16, "id": "6d73e622", "metadata": {}, "outputs": [], "source": [ "def print_element_list(element_list):\n", " print('[')\n", " for element in element_list:\n", " print_element(element)\n", " print(']')" ] }, { "cell_type": "code", "execution_count": 17, "id": "1f64d3bd", "metadata": { "tags": [ "fill-in" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[\n", "Tag\n", "NavigableString\n", "Tag\n", "]\n" ] } ], "source": [ "print_element_list(element.contents)" ] }, { "cell_type": "markdown", "id": "d08fbd7f", "metadata": {}, "source": [ "Now let's try navigating the tree. I'll start with the first child of `element`." ] }, { "cell_type": "code", "execution_count": 18, "id": "c24c19c2", "metadata": { "tags": [ "fill-in" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Tag\n" ] } ], "source": [ "child = element.contents[0]\n", "print_element(child)" ] }, { "cell_type": "markdown", "id": "5fb945ad", "metadata": {}, "source": [ "And print its children." ] }, { "cell_type": "code", "execution_count": 19, "id": "099c0497", "metadata": { "tags": [ "fill-in" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[\n", "Tag