10. IO and Exceptions

Introduction

In this module we will explore basic input/output (I/O) operations in Python, as well as introducing related error and exception handling techniques, including:

• I/O Basics - opening files, stream objects, binary vs text files, reading and writing files, bytearray objects and associated common methods
• Exception Handling - predefined exceptions, try-except-else-finally block, exception hierarchies, assertions and the anatomy of an exception object

The source code for this module may be found in the public GitHub repository for this course. Where code snippets are provided in this module, you are strongly encouraged to type and execute these Python statements in your own Jupyter notebook instance.

1. IO Basics

Most software systems, whether written in Python or not, usually need to read data from one or more data sources and/or need to permanently write data to one or more storage mediums at some point in their logical architecture. Input/Output (I/O) refers to the physical communication between different components in a single system, and indeed across different systems. In this module, we will explore how Python can be used to open, read and write both text-based and binary files using a local filesystem.

1.1. Files and Serialization

The files found in a filesystem are simply containers used to permanently store related information persisted to a non-volatile storage medium, for example a hard disk. In this definition, non-volatile refers to the ability for hard drives, and similar non-volatile storage mediums, to persist the data in the files even when the computer is turned off. When we restart computers, we expect our files, and the information stored within them, to be available to us immediately.

Conversely, the Random Access Memory (RAM) in our computers is termed volatile, as the information stored within memory is not persisted between system restarts. Since Python, via the Python Memory Manager, manages Python objects in memory, if we restart the Python interpreter or the computer then those objects will be lost and we will need to re-run our Python applications in order to re-create them. As such, we use files to persist objects from memory. The process of writing objects from volatile storage mediums (such as RAM) to non-volatile storage mediums (such as local hard-drives) is called serialization, where the object in memory is transformed into a stream of bytes so that it can be stored on disk or transmitted over a computer network. Deserialization on the other hand refers to the reverse prosess where a stream of bytes is transformed into an object in memory.

1.2. Opening Files

The Python standard library provides the open(file, mode) function to allow us to easily open files. To open files using the open() function, we pass two arguments - the relative or absolute path to the file that we wish to open, and an optional mode in which to open the file, which is a string argument with the following options:

• "r" - open a given file for reading (default), returning an error if the file does not exist.
• "a" - open a given file for appending to, which will also create the file if it does not exist.
• "w" - open a given file for writing, which will also create the file if it does not exist.
• "x" - creates the given file on the filesystem, returning an error if the file already exists.
• "t" - handle the given file as text-based (default), meaning that string objects are used to read from and write to the file. The strings are encoded using a given encoding (passed via the encoding argument to the open() function, for example encoding=utf-8) - if no encoding is given then the current platform-dependent locale encoding is used, which by default is utf-8 on Unix-like systems but cp-1252 on Windows.
• "b" - handle the given file as a binary file, meaning that data is read from and written to the file as byte objects. This mode is used for non text-based files such as images, videos, audio and software executable files.

The open() function returns a file object. In sections 1.3 and 1.4 below, we demonstrate how to use the Python open() function to read and write files respectively.

In the following examples, we use the full-text version of the book Alice in Wonderland to demonstrate opening and reading files. You can dowload the full-text version of this classic book from the Project Gutenberg website, which is an online library of over 60,000 free e-books.

1.3. Reading Files

To read a file in Python, we first invoke the Python open() function in read-mode (which is the default mode). To then read the entire contents of the file, we invoke the read() method on the file object. This method will return a string in text mode (default) and bytes object in binary mode. Furthermore in text mode, platform-specific line endings will be translated into \n. In Unix-like operating systems, line-endings already use the line feed control character \n. However in Windows, line-endings use the carriage return and line feed control character sequence \r\n, so it important to note that read() will translate this control character sequence to \n. Once the end of a text file is reached, read() will return an empty string '', as follows:

# Open the full Alice in Wonderland text in read-mode using a relative path
text_file = open('data/alice_in_wonderland.txt', 'r', encoding='utf-8')

# Read the entire contents of Alice in Wonderland
text_file.read()

# Try to read the file again. An empty string will be returned as read() has reached the end of the file.
text_file.read()

You may have noticed in the last example that if we try to invoke the read() method on the same file object once it has already been read entirely, an empty string is returned. We cannot re-read the entire file again using just this method. Computers read files using a file pointer, or cursor, which keeps track of our current location in the file. Specifically it tracks the next byte to be read, and when a file is opened for the first time, the file pointer is placed at the beginning of the file. Each read (and write) operation moves the pointer by an offset relative to the start of the file, which is the number of bytes being read or written in that operation. Once the end of the file is reached, there are no further bytes to read and hence an empty string is returned (in text mode).

We can invoke the tell() method on the file object that will return an integer representing the current position of the file pointer, that is the number of bytes from the beginning of the file in binary mode, and the number of characters in text mode. We can also invoke the seek(offset, whence) method on the file object to move the file pointer to a given position in the file, relative to the whence argument. If this argument is 0 (default) then the offset is relative to the beginning of the file. A value of 1 is relative to the current position of the pointer, and a value of 2 is relative to the end of the file. Finally, we can pass a size argument to the read() method to read a specific number of characters (in text mode) or a specific number of bytes (in binary mode) relative to the current position of the file pointer (when we do not pass a size argument to the read() method, the default behaviour is to read the entire contents of the file as we have seen), as follows:

# Get the current position of the file pointer
print(text_file.tell())

# Move the file pointer to the start of the file
text_file.seek(0)

# Get the current position of the file pointer
print(text_file.tell())

# Read the next 100 characters relative to the current position of the file pointer
text_file.read(100)

# Get the current position of the file pointer
print(text_file.tell())

# Read the next 100 characters relative to the current position of the file pointer
text_file.read(100)

# Read the rest of the file
text_file.read()

We can also use the readline() method to read one line, again relative to the current position of the file pointer. The readline() method identifies the end of a line by the newline character, and will return the line as a string along with the line feed control character \n at the end. Note that if readline() returns an empty string, then the end of the file has been reached, whereas a blank line within the text is returned as a string containing only the \n character.

# Move the file pointer to the start of the file
text_file.seek(0)

# Read one line
text_file.readline()

# Read the next line
text_file.readline()

# Read the next line
text_file.readline()

Finally, another trivial way to read the entire contents of a file is to iterate over each line of the file using a for loop, as follows:

# Move the file pointer to the start of the file
text_file.seek(0)

# Read the entire contents of a file with a for loop
for line in text_file:
    print(line)

1.4. Closing Files

We should explicitly close files once we have finished all relevant operations on the file object. Closing files has the effect of releasing the system resources that were required to manage that file, as well as allowing other external processes to commit any changes to that file if required. Note that the Python memory manager will periodically run a garbage collector which will destroy any objects no longer required by your Python programs, including file objects, however this should not be relied upon. To close a file in Python, we simply call the close() method on the file object. If we attempt to read a file that has been closed, a ValueError exception will be raised, as follows:

# Close the Alice in Wonderland file
text_file.close()

# Try to read the text file after it has been closed
text_file.read()

1.5. Writing Files

To write to a file in Python, we first invoke the Python open() function in either write-mode 'w', append-mode 'a' or exclusive creation mode 'x'. Both write-mode and append-mode are used to write to existing files, but the key difference is that in write-mode, the contents of the file will be ovewritten if the file already exists, whereas in append-mode the file will be appended to. In both cases, if the files does not already exist then the file is created. Exclusive creation mode is used to explicitly create a file with the knowledge that it does not already exist - if it does, an error is returned.

After calling the open() function with the correct mode, a file can be written to by invoking the write() method on the file object and passing a sequence of bytes for byte files or a string for text-based files. Note that we need to explicitly include the newline character \n to create new lines. Python will then translate occurences of \n back to the platform-specific line endings as describes above.

# Open a file in write-mode
my_text_file = open('data/example.txt', 'w')

# Write some text to the file
my_text_file.write('First line\n')
my_text_file.write('Second line\n')
my_text_file.write('Third line\n')

# Close the file
my_text_file.close()

1.6. With Keyword

The with keyword provides us with an alternative (and recommended) way to process file objects. If we use the with keyword to open a file, then that file is automatically closed at the end of the indented code block, as follows:

# Open a file in read-mode using the with keyword
with open('data/alice_in_wonderland.txt', 'r') as text_file:
    text_file_contents = text_file.read()

# Check to see if the file has closed
print(text_file.closed)

# Print the string literal assigned to the variable text_file_contents
print(text_file_contents)

# Open a file in write-mode using the with keyword
with open('data/example2.txt', 'w') as text_file:
    text_file.write('Line 1\n')
    text_file.write('Line 2\n')
    text_file.write('Line 3\n')

1.7. Stream Objects

In Python, there are three main types of file objects, namely buffered text files, buffered binary files and raw binary files. A file object in Python, also known as a file-like object or a stream, can fall into either of these categories, and is defined as an object that exposes file-oriented methods to an underlying resource, such as read() and write() methods. Streams, regardless of whether they are text, binary or raw binary, all share common characteristics - they can be read-only, write-only or read-write. Furthermore, the data stored in the stream can be randomly accessed (that is the file pointer can be moved forwards or backwards to any arbitrary location) or sequentially accessed (that is the data is read in order, for example when a stream of data is transmitted over a network socket).

A stream buffer is the object responsible for holding stream data temporarily to improve the performance of I/O operations. Buffered I/O, such as buffered text streams and buffered binary streams, use the buffer to hold blocks of data up to a certain size that is read from or written into the buffer as a block during I/O operations. Line buffering refers to a similar process, but where the data is stored until a newline is read from or written into the buffer. The standard output stream stdout, which the print() function uses by default, is an example of a buffered stream, whereas the standard error stream stderr is as example of an unbuffered stream where data appears at the destination as soon as it is written, and not in blocks.

A good analogy to help visualise the concept of streams and buffers is that of man-made dams built across rivers. A river is a stream of water moving from a source location to a destination. Dams can be used to store water, and once the dam reaches a certain capacity, that water is released in batches to help with water management and in some cases prevent flooding. In this analogy, the water represents the bytes of data in a stream, and the dam represents the buffer. And in the case of I/O, the movement of the water can represent, for example, the streaming of data from a file into memory or vice-versa, or the transmission of data over a network socket. If we were to use normal Python objects to send data over a network, for example a standard string object, the entire string object would need to be constructed at and sent from the source, transmitted, and then received and processed at the destination - where each step would need to process the object in its entirety. Instead streams and buffers allow us to perform these steps in blocks, which is much more efficient and performant, especially for large objects.

We create 'concrete' file objects or streams when we call the open() function, given an underlying file-system based resource and the mode argument. Text streams are created when calling the open() function in text-mode (default) - they are called text streams because they expect to read and write string objects, where encoding of strings to bytes and decoding of bytes to strings is undertaken transparently. We can also create in-memory text streams (i.e. not supported by an underlying file-system resource but rather in-memory text buffers) using the io module and its StringIO class, as follows:

import io

# Create an in-memory text stream and write text to the buffer
output = io.StringIO()
output.write("Text data\n")
output.write("More text data")

# Print the entire contents of the in-memory text stream buffer using the stdout output stream
contents = output.getvalue()
print(contents)

# Close the in-memory text stream and discard the buffer
output.close()

Similarly, binary streams are created when calling the open() function in binary mode 'b' - they are called binary streams because they expect to read 'bytes-like objects' (i.e. objects that work with binary data including bytearray objects used for compression, saving to a binary file, and transmitting binary over a network socket amongst other use cases) and they expect to write bytes object. With binary streams, no encoding, decoding or newline translation is performed as we are not directly working with string objects but rather bytes literals instead. We can also create in-memory binary streams that use an in-memory bytes buffer via the io module and its BytesIO class as follows:

import io

# Create an in-memory binary stream and write binary data to the buffer
output = io.BytesIO(b"Initial binary data \xf6 \N{MOUSE} \u2708")

# Get a mutable view of the contents of the buffer without copying it
buffer_view = output.getbuffer()
buffer_view[0:10]

# Get a bytes literal containing the entire contents of the buffer
buffer_bytes_contents = output.getvalue()
buffer_bytes_contents

Finally raw binary streams are unbuffered, meaning that binary data appears at the destination as soon as it is written (for output streams). Raw binary streams are rarely used because we often wish to manipulate data in its high-level primitive form (for example string objects or bytes objects). Nevertheless, raw binary streams may be created when calling the open() function in binary mode 'b' and passing an argument to disable the buffer, as follows:

# Create a raw binary stream
raw = open('my_file.ext', 'rb', buffering=0)

1.8. Bytearray Objects

In the previous modules of this course, we have explored strings, lists, tuples and range objects that are all used to store sequences of data. There are two further objects in Python that support sequence data, namely bytes objects and bytearray objects.

Both of these object types store sequences of single bytes, where bytes objects are immutable sequences and bytearray objects are mutable sequences. Bytes objects may be created from byte literals, where a byte literal is defined using a b prefix and ASCII characters, where each ASCII character has an equivalent binary value. As such, bytes objects are in fact sequences of integers representing binary values. Where binary values for characters exceed 127 (i.e. the maximum binary value available in the ASCII character set), the appropriate escape sequence must be used in the byte literal, as follows:

# Create a byte literal that returns a byte object
bytes_objects = b'my byte literal'

# Get the binary value for the ASCII character at position 1 in the byte sequence
# i.e. the ASCII character 'y' has a binary value of 121
bytes_objects[1]

# Try to create a byte literal with a character that has a binary value greater than 127
invalid_bytes_object = b'my invalid byte literal ö'

# Instead we must use the equivalent escape sequence
valid_bytes_object = b'my valid byte literal \xf6'
print(valid_bytes_object[1])
print(valid_bytes_object[-1])

Bytearray objects on the other hand are created using the bytearray() constructor function, and are mutable arrays of given bytes. They may be created as an empty instance (containing no byes), a zero-filled instance of a given length, from an iterable of integers, or by copying existing binary data, as follows:

# Create an empty bytearray
my_empty_bytearray = bytearray()
my_empty_bytearray

# Create a zero-filled bytearray of a given length
my_zero_filled_bytearray = bytearray(10)
my_zero_filled_bytearray

# Create a bytearray from a given iterable of integers
my_bytearray_from_iterable = bytearray(range(0, 20))
my_bytearray_from_iterable

# Create a bytearray by copying existing binary data
my_bytearray_from_binary_data = bytearray(b'my valid byte literal \xf6')
my_bytearray_from_binary_data

Since both bytes and bytearray objects are sequences of data (i.e. bytes), they support the same common operations that the other sequence types support, including slice notation, the + operator to concatenate sequences, the in and not in operators to test for membership, the len(), min() and max() functions, and the index() and count() methods respectively. Note however that since bytes and bytearrays objects are sequences of integers, b[n] will return an integer representing the binary value of the byte at that specific index position, whilst b[x:y] will return a bytes object of the associated length. This is different to string objects where both indexing and slicing will return a string object.

# Demonstrate common sequence operations using the bytes object type

# Create a bytes object
my_bytes_object = b'El Ni\xc3\xb1o'

# Convert a bytes object to a string object
my_bytes_string = my_bytes_object.decode()
print(my_bytes_string)

# Calculate the length of the bytes object, bearing in mind that the UTF-8 character code for ñ is two bytes
print(len(my_bytes_object))

# Extract the integer value of a specific byte using its zero-indexed index position
print(my_bytes_object[0]) # Binary value for the ASCII character at index 0 (i.e. e)
print(my_bytes_object[1]) # Binary value for the ASCII character at index 1 (i.e. l)
print(my_bytes_object[2]) # Binary value for the ASCII character at index 2 (i.e. space)
print(my_bytes_object[5]) # Binary value for \xc3, where \xc3\xb1 is the UTF-8 character code for ñ
print(my_bytes_object[6]) # Binary value for \xb1, where \xc3\xb1 is the UTF-8 character code for ñ
print(type(my_bytes_object[1]))

# Extract a subset of the bytes (as a bytes object) from the bytes object using slice notation
print(my_bytes_object[0:3])
print(my_bytes_object[5:-1])
print(type(my_bytes_object[0:3]))

# Return the character represented by the integer value for a given byte
print(chr(my_bytes_object[1]))
print(chr(my_bytes_object[5]))
print(chr(my_bytes_object[6]))

# Concatenate two bytes objects to create a new bytes object
my_concatenated_bytes_object = b'ABC' + b'123'
print(my_concatenated_bytes_object)
print(my_concatenated_bytes_object[0])
print(my_concatenated_bytes_object[3])
print(my_concatenated_bytes_object[0:3])
print(my_concatenated_bytes_object[3:6])

# Membership operator
print(65 in my_concatenated_bytes_object)
print(49 in my_concatenated_bytes_object)
print(b'ABC' in my_concatenated_bytes_object)
print(b'XYZ' in my_concatenated_bytes_object)
print(b'3' in my_concatenated_bytes_object)

# Count method
my_new_bytes_object = b'abracadabra'
print(my_new_bytes_object.count(b'a'))
print(my_new_bytes_object.count(b'b'))

# Index method
print(my_concatenated_bytes_object.index(49))
print(my_concatenated_bytes_object.index(b'1'))

1.9. Readinto Method

We now know that bytearray objects are mutable sequences of given bytes. We can use bytearray objects to create a writable data buffer by invoking the bytearray() constructor and passing to it the entire bytes object returned by file.read() as follows:

# Create a bytearray object using the bytes object returned from the read() method
binary_stream = open('data/alice_in_wonderland.txt', 'rb')
bytearray_data = bytearray(binary_stream.read())

# Print the integer value of a given byte in the bytearray
print(bytearray_data[1])

# Close the binary stream
binary_stream.close()

However a more efficient means to read into a bytearray is to use the readinto() method, which reads bytes into a pre-allocated (i.e. pre-sized) bytearray and then returns the number of bytes read, as follows:

import os

# Create a binary stream from opening a file in binary mode
binary_stream = open('data/alice_in_wonderland.txt', 'rb')

# Get the size of the file in bytes
file_size = os.path.getsize('data/alice_in_wonderland.txt')
print(file_size)

# Create a bytearray of pre-defined size i.e. the file size in bytes
bytearray_data = bytearray(file_size)

# Read bytes from the binary input stream into the pre-allocated bytearray
binary_stream.readinto(bytearray_data)

# Print the length of the bytearray, which should match the file size
print(len(bytearray_data))

# Close the binary stream
binary_stream.close()

1.10. Common File Methods

Now that we have an understanding of streams and buffers in Python, we can re-conceptualise the Python open() function. When we open a given file in text-mode, we are in fact instantiating a text stream of string objects where encoding of string to bytes (output stream) and decoding of bytes to strings (input stream) is undertaken transparently. Similarly, when we open a given file in binary-mode, we are in fact instantiating a stream of bytes where we read bytearrays objects (input stream) and write bytes objects (output stream). Given this augmented understanding, we can now fully understand the following non-exhaustive methods available to a file object in Python, as follows:

• file.close() - closes an opened file.
• file.detach() - separates the binary buffer from the text buffer and returns it.
• file.flush() - flush the write buffer of the file stream.
• file.isatty() - returns True if the file stream is interactive, False otherwise .
• file.readable() - returns True if the file stream can be read, False otherwise.
• file.readlines() - returns a list containing each line in the file as a list item.
• file.seekable - returns True if the file stream supports random access (i.e. changing the file pointer to an arbitrary location).
• file.writable() - returns True if the file stream can be written to, False otherwise.

# Open the full Alice in Wonderland text in binary read-mode
binary_stream = open('data/alice_in_wonderland.txt', 'rb')

# isatty()
print(binary_stream.isatty())

# readable()
print(binary_stream.readable())

# seekable()
print(binary_stream.seekable())

# writable()
print(binary_stream.writable())

# Close the binary stream
binary_stream.close()

# Open the full Alice in Wonderland text in text read-mode
text_stream = open('data/alice_in_wonderland.txt', 'r', encoding='utf-8')

# Get the underlying binary stream
underlying_binary_stream = text_stream.detach()

# Read and return a given number of bytes from the buffer
print(underlying_binary_stream.read(100))

# Close the binary stream
underlying_binary_stream.close()

2. Exceptions

At a high-level, there are two types of errors inherent to all programming languages. These are syntax errors and runtime errors or exceptions. Syntax errors, also called parsing errors, are errors as a result of writing programming statements that do not conform to the grammar and/or structure of the programming language in question. Fortunately, nearly all modern integrated development environments (IDE) come embedded with automated syntax checkers and autocomplete functionality so that syntax errors are detected in real-time as we write our code. Unfortunately Jupyter Notebook does not come embedded with such syntax errors or autocomplete functionality as standard, but the problem is less severe when developing code in notebooks because (a) notebooks are primarily designed for non-production code development, exploratory analysis and prototyping, and (b) code in notebooks are executed a cell at a time, meaning that syntax errors are caught quickly.

The following code snippets are all examples of syntactically invalid Python statements which will cause the Python interpreter to return a SyntaxError, or an IndentationError in the case of code blocks that are incorrectly indented:

# Examples of syntactically invalid Python statements
print 'Legacy print syntax found in Python 2.x'

# Invalid if conditional statement (missing colon)
x = 1
if x > 0
    print('x is larger than 0')

# Invalid if conditional statements (incorrect indentation)
x = 1
if x > 0:
print('x is larger than 0')

# Invalid identifier
^a = 3

# Invalid literal
my_literal = w'literal'

As you become more experienced in a given programming language, and especially when developing your code in a modern IDE, you naturally make less syntax errors. And even if you do, the IDE will automatically detect it, explain why it is syntactically incorrect, and provide autocomplete options thereafter. However, even if your code is free of syntax errors, other errors may be encountered.

These other errors are called runtime errors or exceptions. They occur when a certain condition during runtime causes a logical error in your syntactically correct code. Example conditions include dividing by zero, the non-existence of a file when attempting to open it using the open() function in read-mode, and importing a 3rd party module that did exist in the Python development virtual environment but does not exist in the Python production virtual environment. These types of errors detected during runtime are called exceptions. By default when an exception is encountered, the specific type of exception is raised and your Python application will terminate. Fortunately we will soon learn how to handle exceptions so that they are not fatal to our running applications.

2.1. Predefined Exceptions

Exceptions come in different types dependent on the logical error that caused them to be raised. For example, dividing by zero will raise a ZeroDvisionError exception, trying to open a file in read-mode that does not exist will raise a FileNotFoundError exception, and importing a module that does not exist will raise a ImportError exception, as follows:

# ZeroDivisionError exception
print(100/0)

# FileNotFoundError exception
my_non_existent_file = open('/i/do/not/exist.txt', 'r')

# ImportError exception
import nonexistentmodule

These are all examples of built-in exceptions, where the exception names are actually built-in identifiers in Python. The error message displayed is made up of a stacktrace providing contextual details of where the exception was encountered, the exception name itself, and then a string description of what caused the exception. Fortunately the Python standard library provides numerous built-in exceptions that cover a wide range of logical errors such as those provided above, and including but not limited to:

• IndexError - when a given index of a sequence is out of range.
• KeyError - when a dictionary key is not found.
• MemoryError - when an operation runs out of memory.
• NameError - when a variable is not found in the local or global scope.
• TypeError - when a function, method or operation is applied to an object of incorrect type.
• ValueError - when a function is passed an argument of the correct type but invalid value.
• RuntimeError - when a logical error is encountered that does not fall into any other category.

2.2. Exception Hierarchies

In Python, exceptions are in fact instances of a class where the type of class is denoted by the exception name that is raised. Hence exceptions themselves are objects. All exceptions in Python are derived from the BaseException class and follow a hierarchy. This means that if we explictly handle an exception for a particular exception class type, we are implicitly handling all exception classes derived from that class. The class hierarchy for built-in exceptions in Python is as follows:

BaseException
 +-- SystemExit
 +-- KeyboardInterrupt
 +-- GeneratorExit
 +-- Exception
      +-- StopIteration
      +-- StopAsyncIteration
      +-- ArithmeticError
      |    +-- FloatingPointError
      |    +-- OverflowError
      |    +-- ZeroDivisionError
      +-- AssertionError
      +-- AttributeError
      +-- BufferError
      +-- EOFError
      +-- ImportError
      |    +-- ModuleNotFoundError
      +-- LookupError
      |    +-- IndexError
      |    +-- KeyError
      +-- MemoryError
      +-- NameError
      |    +-- UnboundLocalError
      +-- OSError
      |    +-- BlockingIOError
      |    +-- ChildProcessError
      |    +-- ConnectionError
      |    |    +-- BrokenPipeError
      |    |    +-- ConnectionAbortedError
      |    |    +-- ConnectionRefusedError
      |    |    +-- ConnectionResetError
      |    +-- FileExistsError
      |    +-- FileNotFoundError
      |    +-- InterruptedError
      |    +-- IsADirectoryError
      |    +-- NotADirectoryError
      |    +-- PermissionError
      |    +-- ProcessLookupError
      |    +-- TimeoutError
      +-- ReferenceError
      +-- RuntimeError
      |    +-- NotImplementedError
      |    +-- RecursionError
      +-- SyntaxError
      |    +-- IndentationError
      |         +-- TabError
      +-- SystemError
      +-- TypeError
      +-- ValueError
      |    +-- UnicodeError
      |         +-- UnicodeDecodeError
      |         +-- UnicodeEncodeError
      |         +-- UnicodeTranslateError
      +-- Warning
           +-- DeprecationWarning
           +-- PendingDeprecationWarning
           +-- RuntimeWarning
           +-- SyntaxWarning
           +-- UserWarning
           +-- FutureWarning
           +-- ImportWarning
           +-- UnicodeWarning
           +-- BytesWarning
           +-- ResourceWarning

For example, if we explicitly handle exceptions of type Exception, we are implicitly handling nearly all the built-in exception types in Python.

In section 2.5. User Defined Exceptions of this module, we will study how we can create and raise custom exceptions by creating a new exception class. When we create custom exception classes, it is recommended that they are derived from the Exception class or indirectly via one of its subclasses, and not from BaseException.

2.3. Try Except Else Finally

Fortunately Python, like most other programming languages, provides a means to handle exceptions. If we do not handle exceptions, then our Python applications may fail under unforeseen conditions. This is particularly frustrating for long-running or mission-critical Python applications which, upon termination due to an unhandled exception, may then need to be diagnosed, refactored, re-tested, re-deployed and re-run, potentially causing lengthy and costly delays.

It is unrealistic to be able to predict and subsequently handle all logical errors, especially in complex systems with many dependencies and for those logical errors not covered by built-in Python exceptions. The advantage of modern continuous integration and continuous deployment (CI/CD) frameworks, tools and pipelines is that code releases can be automatically tested, integrated and deployed into production environments extremely quickly (in some cases, hundreds or even thousands of code releases can be deployed into production every day), and triggered by code commits and pushes in modern software version control systems such as Git. It is therefore strongly recommended that when developing code, even at the prototype or proof-of-concept stage, that you provision adequate CI/CD pipelines to reduce the costs and risks of refactoring code to handle unforeseen exceptions.

Python provides the try-except-else-finally statements that allow us to:

• try - execute an indented block of code, and listen for logical errors.
• exception - catch any exceptions that are raised, or specific exceptions, with a subsequent indented block of code defining the operations required to handle the exception specific to our application.
• else - optional indented code block following exception that defines operations that will run only if no exceptions are raised.
• finally - execute an optional indented block of code in all cases i.e. whether an exception is raised or not.

2.3.1. Try Except

In the following example, we use the try-except statements to handle exceptions raised as a result of file stream operations. First the statements between the try and except clauses are executed. If no exception is raised, the except clause is skipped and the execution of the try statement is finished. However if an exception is raised during execution of the statements in the try clause, then the try clause will stop as soon as the exception is encountered. Thereafter, if the raised exception type matches the exception type named after the except keyword, then the except clause is executed.

# Try to open a text file in read-mode that does not exist
try:
    with open('idonotexist.txt', 'r') as my_file:
        data = my_file.read()
except:
    print('An error was encountered')

print('I will still be printed')

In the example above, we use a bare except statement (that is the except keyword followed by no exception type) that will catch all exception types. Whilst this will allow our Python application to continue to run, by using a bare except statement we will never know the specific exception type and hence specific logical error that caused our code to fail. Therefore it is not recommended to use a bare except statement. Instead we should explicitly specify the exception type that we wish to catch and handle, bearing in mind the hierarchy of exceptions types, as follows:

# Try to open a text file in read-mode that does not exist
try:
    with open('idonotexist.txt', 'r') as my_file:
        data = my_file.read()
except Exception as e:
    print(f'An error was encountered:\n{e}')
    print(type(e))
    print(e.__class__)

print('I will still be printed')

In this example, we explicitly catch and handle exceptions of the type Exception which, as a result of the built-in exception hierarchy, will also catch all the exception types derived from it. We also use the as keyword to create an alias for our exception object, which we can then use to display the string description of what caused the exception. Finally, because we are explicitly handling exceptions of the type Exception which has many exception types derived from it, we can identify the exact exception type that caused the error by using the type() function given the exception object, or equivalently the __class__ attribute. In this case, the exception raised was of type FileNotFoundError.

2.3.2. Unhandled Exceptions

We now know that the except clause is executed only if the raised exception type matches the exception type named after the except keyword. But what happens if an exception is raised but we do not declare a matching except clause? In this case, Python will pass the exception to outer try statements to find a matching handler. If, as a result of traversing the outer statements in your application, no matching handler is found to deal with the raised exception type, the Python application will terminate and display the exception and error message. This is known as an unhandled exception.

2.3.3. Outer Try Statements

If no matching handler is found to deal with a raised exception type in the current try statement, then Python will pass the exception to outer try statements to find a matching handler. Usually, outer try statements would be defined in a higher-level function, as follows:

import math

# Pass a raised exception to matching handler in an outer try statement defined in a higher-level function
def my_function(x):
    try:
        return square_root(x)
    except Exception as e:
        print(f'An error was encountered of type: {type(e)}')
        print(e)

def square_root(n):
    try:
        return math.sqrt(n)
    except TypeError as e:
        print('Please provide a real number.')

# Call the higher-level function with a string argument which will raise a TypeError
my_function("I am a string")

# Call the higher-level function with a negative number which will raise a ValueError not handled by the square_root() try statement
my_function(-1)

In the example above, we define a function called square_root() that will calculate the square root of a given number within a try statement. It will also catch exceptions of the type TypeError that are raised should the given argument n not be a real number. We then define a higher-level function called my_function() that will call our square_root() function within its own try statement (i.e. this is the outer try statement) with a given argument x. Our higher-level function will catch exceptions of the type Exception and, via exception hierarchy, all exception types derived from Exception.

When we call my_function() with a string argument, math.sqrt(str) will raise a TypeError that is handled within the square_root() function. However when we call my_function() with an invalid real number argument (in this case a negative number), math.sqrt(-n) will raise a ValueError that is not handled by the square_root() function. In this case, the ValueError exception will be passed to the outer try statement defined in my_function() where there is a matching handler (as ValueError is derived from Exception).

2.3.4. Try Except Finally

In the following example, we introduce the optional finally clause that executes an indented block of code regardless of whether an exception is raised or not. In general, statements in the finally clause are used to clean and free up system resources, as follows:

# Finally is generally used to clean up and free system resources
try:
    my_writable_file = open('data/my_writable_file.txt', 'w')
    my_writable_file.write('Line 1')
    my_writable_file.write(100)
except Exception as e:
    print(f'An error was encountered:\n{e}')
finally:
    my_writable_file.close()

In the example above, my_writable_file.write(100) will raise a TypeError since the write() method expects a string argument. The statement my_writable_file.close() in the finally clause will always run, regardless of whether an exception is raised in the try statement or nto. In this case, it will close the file thereby releasing the system resources that were required to manage that file in memory.

2.3.5. Try Except Else

In the following example, we introduce the optional else clause that executes an indented block of code only if the try clause does not raise an exception, as follows:

# Execute code only if the try clause does not raise an exception
while True:
    try:
        x = int(input('Please enter a number: '))
    except ValueError:
        print("Invalid argument.")
    else:
        if x % 2 == 0:
            print('Your number is even.')
        else:
            print('Your number is odd.')
        break

In the example above, assuming a valid numerical value is provided by the user, the statements in the else clause will run and the while loop will terminate thereafter. If an invalid value is provided by the user, for example a string literal, then a ValueError exception will be raised and the while loop will cause the program to continue indefinitely until a valid value is provided.

2.3.6. Multiple Exceptions

A try clause can have multiple except clauses, each designed to handle different exceptions, or groups of exceptions. However only one exception handler will ever be executed. Also note that exception handlers will only handle exceptions raised by the statements in the try clause - they do not handle exceptions raised by the statements in other except clauses. The following example demonstrates how a try statement may have more than once except clause:

# Handle multiple possible exceptions
def square_root(n):
    try:
        return math.sqrt(n)
    except TypeError:
        print('Please provide a real number.')
    except ValueError:
        print('Please provide a positive real number.')

# Invoke the TypeError exception handler
square_root('one')

# Invoke the ValueError exception handler
square_root(-1)

Furthermore a single except clause may be defined to handle multiple exceptions by providing a tuple of exception types, as follows:

# Define a single except clause to handle multiple exception types
def reciprocal(n):
    try:
        return 1 / n
    except (TypeError, ValueError):
        print('Please provide a valid real number.')
    except (ZeroDivisionError):
        print('The reciprocal of zero is undefined.')

# Invoke the first except handler
reciprocal('one')

# Invoke the second except handler
reciprocal(0)

2.4. Raising Exceptions

In addition to handling exceptions, many programming languages allow developers to explicitly force a specified exception. In Python this is achieved using the raise keyword followed by an exception class (derived from Exception) which is then automatically instantiated, or an exception instance, as follows:

# Explicitly raise an exception
def square_root(n):
    if n < 0:
        raise ValueError('Only positive integers can have square roots.')
    else:
        return math.sqrt(n)

# Call this function and try to calculate the square root of a negative number
square_root(-1)

2.5. User Defined Exceptions

As we have seen, the Python standard library provides a wide number of built-in exceptions, where a RuntimeError exception is raised when the logical error does not fall into any other category. However since an exception class is just like any other class in Python, there is nothing stopping us from creating, using and distributing our own exception classes, including overriding the constructor of the built-in Exception class through inheritance.

When we create user-defined exception classes in Python specific to our applications and use-cases, normally we start by defining a custom exception base class that is derived from the built-in Exception class. Thereafter we can define other exception classes that are derived from our custom base class. In the following example, we create a series of custom exception classes starting with a base class that is derived from Exception, and two other custom exception classes that are derived from our custom base class:

# Create a custom exception for a card in a deck of playing cards
class CardError(Exception):

    def __init__(self, message='Invalid playing card.'):
        self.message = message
        super().__init__(self.message)


# Create a custom exception for an invalid playing card value
class CardValueError(CardError):
    """Exception raised when an invalid playing card value is encountered.

    Attributes:
        value (str): card value, where valid values are ACE, 2 - 10, JACK, QUEEN, KING
    """

    def __init__(self, value):
        self.value = value
        self.message = f'{value} is an invalid playing card value'
        super().__init__(self.message)


# Create a custom exception for an invalid playing card suit
class CardSuitError(CardError):
    """Exception raised when an invalid playing card suit is encountered.

    Attributes:
        suit (str): card suit, from HEARTS, DIAMONDS, SPADES, CLUBS
    """

    def __init__(self, suit):
        self.suit = suit
        self.message = f'{suit} is an invalid playing card suit'
        super().__init__(self.message)

# Enter an invalid playing card value and raise a CardValueError exception
card_value = input('Please enter a playing card value: ')
if card_value.upper() not in ['ACE', '2', '3', '4', '5', '6', '7', '8', '9', '10', 'JACK', 'QUEEN', 'KING']:
    raise CardValueError(card_value)

# Enter an invalid playing card suit and raise a CardSuitError exception
card_suit = input('Please enter a playing card suit: ')
if card_suit.upper() not in ['HEARTS', 'DIAMONDS', 'SPADES', 'CLUBS']:
    raise CardSuitError(card_suit)

In the example above, we have created a series of custom exception classes that are designed to be raised when an invalid playing card is encountered. We start with our custom exception base class called CardError which is derived from the built-in Exception class and simply calls the constructor of the Exception superclass with a bespoke message to display.

Next we define a custom exception class called CardValueError which is derived from our custom CardError exception base class. CardValueError is designed to be raised when an invalid playing card value is encountered i.e. a given playing card object does not have a value in Ace, 2 - 10, Jack, Queen nor King. If an invalid playing card value is encountered, this exception class simply calls the constructor of its superclass i.e. CardError with a bespoke message to display.

Finally, we define another custom exception class called CardSuitError which is also derived from our custom CardError exception base class. CardSuitError is designed to be raised when an invalid playing card suit is encountered i.e. a given playing card object does not have a suit in HEARTS, DIAMONDS, SPADES nor CLUBS. If an invalid playing card suit is encountered, this exception class simply calls the constructor of its superclass i.e. CardError with a bespoke message to display.

By convention, the names of user-defined exception classes end in Error similar to the built-in exception classes, such as CardError, CardValueError and CardSuitError in the previous example.

2.6. Assertions

The final Python topic that we will cover in this module, and indeed this course, is that of assertions. In Python, the assert keyword allows us to test whether a given condition is True, else an AssertionError exception will be raised. Assertions form the basis of debugging, unit testing and subsequent test-driven development (TDD) in many languages including Python. Any valid Python statement may be used in an assert statement, where the emphasis is on asserting that a given condition is True, as follows:

# Use assert to ensure that a given argument for the square root function is valid
def square_root(n):
    assert isinstance(n, int), "The given argument is not an integer"
    assert n >= 0, "The given integer is negative"
    return math.sqrt(n)

# Call the square_root() function and try to calculate the square root of a string object
square_root('str')

# Call the square_root() function and try to calculate the square root of a negative number
square_root(-1)

In the example above, we use assert to test whether the given argument to our square_root() function is a positive integer. If not, an AssertionError will be raised. Note that the string literal following the condition in the assert statement is optional, and is used to display a custom error message if that assertion is not True.

Summary

In this module we have explored basic I/O in Python, including buffered text I/O, buffered binary I/O and unbuffered raw binary I/O. We now know how to open files and perform standard file operations including reading from, writing to and closing files. We also have an in-depth understanding of exception handling in Python, including exception hierarchies and how to create and use our own user-defined exception classes.

Homework

Please write Python programs for the following exercises. There may be many ways to solve the challenges below - first focus on writing a working Python program, and thereafter look for ways to make it more efficient using the techniques discussed both in this module and over this course thus far.

1. Reading Files - Dictionary of World Countries
   In the GitHub repository for this course, there exists a CSV lookup of world countries and associated metadata including capital city, capital city latitude, captial city longitude and country code. This CSV lookup may be found in data/country-capitals.csv. Write a Python program that reads and parses this CSV into a dictionary object where the dictionary keys are the unique country names, and the dictionary values are tuples containing the associated metadata.
   
   
2. User-Defined Exceptions - Missing Capital Cities
   If you take a closer look at data/country-capitals.csv, you will notice that there are entries for Antarctica (line 241) and Heard Island and McDonald Islands (line 243) which have no capital city (string literal 'N/A') and hence no capital latitude nor longitude co-ordinates (0 and 0.000000 respectively). Extend your answer to question 1 such that if a record in the lookup has a capital city with a string literal value of 'N/A' then a custom CapitalCityError exception is raised but also handled such that this record is not inserted into the dictionary, but the user is notified of each such invalid record via the print() function. Finally check the number of key:value pairs in your final dictionary to make sure that these two records have not been inserted.

What's Next

In the next module, we will consolidate everything that we have learnt from all the modules in this course by taking a practice Certified Associate in Python Programming (PCAP) examination paper.