8.3 File Input/Output and Context Managers

File Input/Output and Context Managers

Input and output (I/O) operations allow Python programs to interact with the underlying operating system and file storage. In Python 3, file handling is built on top of the io module hierarchy, providing a unified stream abstraction. Mastering stream access modes, character encodings, positioning mechanics (seek/tell), context managers, and exception hierarchies is essential for the PCAP certification.

---

1. The open() Built-in Function and Stream Modes

The primary entry point for file operations in Python is the open() built-in function:

open(file, mode='r', buffering=-1, encoding=None, errors=None, newline=None, closefd=True, opener=None)

Mode Strings: Access Modes and Format Modes

A file mode string is constructed by combining an Access Mode with a Format Mode:

$\text{Full Mode} = \text{Access Mode} + \text{Format Mode}$

A. Access Modes

B. Format Modes: Text ('t') vs Binary ('b')

• 't' (Text Mode, default): Stream operations read and write Python string (str) objects. Automatic character encoding/decoding occurs using the specified (or platform-default) encoding. Universal newline translation automatically converts platform line endings (\r\n on Windows, \n on POSIX) to \n on read, and converts \n to the platform newline on write.
• 'b' (Binary Mode): Stream operations read and write raw bytes objects without character decoding or newline translation. Used for non-text formats (images, audio, compiled binaries, PDFs).

# Default mode is 'rt' (Read Text)
f1 = open("notes.txt", "r")      # Equivalent to open('notes.txt', 'rt')

# Write Binary
f2 = open("image.png", "wb")    # Binary write

# Read & Write Update Mode
f3 = open("data.txt", "r+")     # Opens existing file for reading and writing without truncation

[!WARNING] The 'w+' vs 'r+' Trap: Both 'r+' and 'w+' allow reading and writing. However, 'r+' opens an existing file without truncation, allowing you to overwrite specific sections. In contrast, 'w+' immediately truncates (erases) the file to 0 bytes upon opening.

---

2. Reading Streams: read(), readline(), readlines(), and Line Iteration

Python provides four distinct methods for reading data from an open file stream:

1. file.read(size=-1)

Reads up to size characters (in text mode) or size bytes (in binary mode). If size is negative or omitted, it reads the entire remainder of the file into a single str or bytes object.

with open("sample.txt", "r") as f:
    first_five = f.read(5)   # Reads first 5 characters
    rest_of_file = f.read()  # Reads from character 6 to EOF

2. file.readline(size=-1)

Reads a single line up to and including the trailing newline character \n. If size is specified, it reads at most size characters of the current line.

[!IMPORTANT] Detecting EOF with readline(): When readline() encounters an empty line within a file, it returns "\n" (a string of length 1 containing the newline). It returns "" (an empty string of length 0) strictly when the end of the file (EOF) is reached.

with open("sample.txt", "r") as f:
    while True:
        line = f.readline()
        if line == "":  # Strictly EOF
            break
        print(repr(line))

3. file.readlines(hint=-1)

Reads all remaining lines from the stream and returns them as a list of strings. Every line in the list (except possibly the very last line of a file) retains its trailing \n:

with open("sample.txt", "r") as f:
    lines = f.readlines()
    # ['First line\n', 'Second line\n', 'Third line']

4. Memory-Efficient Line Iteration (for line in file:)

File objects implement the Python Iterator Protocol (__iter__() and __next__()). Iterating directly over the file object streams lines lazily through internal buffers, using $O(1)$ auxiliary memory:

# Recommended idiom for processing large files
with open("server.log", "r") as log_file:
    for line in log_file:
        if "ERROR" in line:
            print(line.strip())

---

3. Writing Streams: write(), writelines(), and Flush Mechanics

file.write(string_or_bytes)

Writes data to the stream and returns the integer number of characters written (in text mode) or bytes written (in binary mode).

with open("output.txt", "w") as f:
    chars_written = f.write("Hello, Python!\n")
    print(chars_written)  # 15

[!NOTE] Unlike print(), file.write() does not append a newline character \n automatically. You must include explicit newlines in the string.

file.writelines(sequence)

Writes an iterable of strings (or bytes) to the file. Despite its name, writelines() does not add newline characters between sequence elements:

items = ["Apple\n", "Banana\n", "Cherry\n"]
with open("fruits.txt", "w") as f:
    f.writelines(items)

Buffering and file.flush()

To optimize disk I/O, Python buffers write operations in memory. Data is physically written to the OS storage when:

1. The internal buffer fills up.
2. file.flush() is called explicitly.
3. The file is closed via file.close() or exiting a with block.

---

4. Stream Positioning: tell() and seek()

Every open file stream maintains an internal stream position pointer.

file.tell()

Returns the current stream position as an integer byte offset from the beginning of the file.

file.seek(offset, whence=0)

Repositions the stream pointer to a new location. The whence parameter specifies the reference point:

# In binary mode ('rb')
with open("binary_data.bin", "rb") as f:
    f.seek(10, 0)   # Jump to byte index 10 from start
    f.seek(5, 1)    # Jump forward 5 bytes from current position (byte 15)
    f.seek(-10, 2)  # Jump to 10 bytes before end of file
    pos = f.tell()  # Check current byte offset

Text Mode seek() Constraints

On the PCAP exam, pay close attention to text mode seek rules:

[!IMPORTANT] In text mode ('t'), arbitrary seeking with non-zero offsets relative to whence=1 (current) or whence=2 (end) is disallowed due to multi-byte character encodings and newline translation. In text mode:

• You may seek with any offset from whence=0 only if offset was returned by a previous tell() call (or 0).
• Seeking relative to whence=1 or whence=2 is only valid with an offset of exactly 0 (e.g., f.seek(0, 2) to jump to the end of a text file).

---

5. Context Managers and the with Statement

Opening files using traditional open() requires explicit cleanup with close(). If an exception occurs between open() and close(), the file descriptor remains open, leaking system resources.

The with Statement Guarantee

The with statement encapsulates execution within a context manager, guaranteeing that resources are deterministically released when execution leaves the block, even if an unhandled exception or return occurs:

# Traditional error-prone pattern
f = open("data.txt", "r")
try:
    content = f.read()
finally:
    f.close()

# Modern idiomatic Python
with open("data.txt", "r") as f:
    content = f.read()
# f is guaranteed to be closed here
print(f.closed)  # True

The Context Management Protocol

An object works with the with statement if it implements the Context Management Protocol:

1. __enter__(self): Executed when entering the with block. Its return value is bound to the target specified after the as keyword.
2. __exit__(self, exc_type, exc_val, exc_tb): Executed when exiting the with block.
   • If no exception occurred, all three arguments are None.
   • If an exception occurred, the exception class, value, and traceback are passed. If __exit__() returns True, the exception is suppressed; if it returns False or None, the exception is re-raised.

class ManagedResource:
    def __enter__(self):
        print("1. Resource allocated")
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        print("3. Resource cleanly closed")
        return False  # Propagate any exceptions

with ManagedResource() as r:
    print("2. Performing work inside block")

---

6. Binary Streams, bytes, and Mutable bytearray

When working with binary files ('rb', 'wb'), Python uses raw byte structures.

bytes vs bytearray

• bytes: An immutable sequence of integers in the range $0 \le x \le 255$. Literal syntax: b'Hello' or bytes([72, 101, 108, 108, 111]).
• bytearray: A mutable sequence of integers in the range $0 \le x \le 255$. Supports item assignment, slicing mutations, .append(), and .extend().

# Creating and mutating a bytearray
ba = bytearray(b"Python")
print(ba)        # bytearray(b'Python')

# Mutate single byte by integer index
ba[0] = ord('J')
print(ba)        # bytearray(b'Jython')

# Slicing and writing to a binary file
with open("binary.dat", "wb") as bf:
    bf.write(ba)

# Reading binary data into a bytearray
with open("binary.dat", "rb") as bf:
    raw_bytes = bf.read()
    mutable_buf = bytearray(raw_bytes)
    mutable_buf.append(33)  # ASCII for '!'
    print(mutable_buf)      # bytearray(b'Jython!')

---

7. The OSError Exception Hierarchy

All I/O-related exceptions in Python 3 inherit from the built-in OSError (which is also aliased as IOError and EnvironmentError for backwards compatibility).

Hierarchy Tree

BaseException
 └── Exception
      └── OSError
           ├── FileNotFoundError       (File does not exist on 'r' mode)
           ├── FileExistsError          (File already exists on 'x' mode)
           ├── PermissionError          (Insufficient OS permissions)
           ├── IsADirectoryError        (Attempted file open on a directory)
           └── UnsupportedOperation     (e.g., write() on 'r' mode stream)

try:
    with open("nonexistent_file.txt", "r") as f:
        data = f.read()
except FileNotFoundError as e:
    print(f"Specific error: {e.strerror} (errno {e.errno})")
except OSError as e:
    print(f"Generic OS error: {e}")