4.1 Character Encodings and String Immutability

Character Encodings and String Immutability

Understanding how Python handles text at the bit and byte level is a cornerstone of the OpenEDG PCAP-31-03 certification. Python 3 introduced a rigorous model that cleanly separates human-readable characters from raw binary sequences.

---

1. Text vs. Bytes in Python 3

In Python 2, strings were handled as raw byte sequences by default, which led to frequent encoding errors when dealing with international character sets. Python 3 resolved this by creating two distinct, non-interchangeable sequence types:

1. str (Text Strings): An immutable sequence of abstract Unicode code points. A str object represents human characters, letters, symbols, and emojis independently of how they are stored as physical bytes in memory.
2. bytes (Byte Sequences): An immutable sequence of raw 8-bit octets (integers in the range 0 through 255). Byte literals use the b prefix (e.g., b'Hello').

# Creating text and byte objects
text_data = "Python π"
byte_data = b"Python"

print(type(text_data))  # <class 'str'>
print(type(byte_data))  # <class 'bytes'>

# Indexing behavior differences
print(text_data[0])     # 'P' (1-character str)
print(byte_data[0])     # 80 (integer representing the ASCII value of 'P')

The Strict Boundary Rule

In Python 3, str and bytes can never be implicitly concatenated or compared for ordered relationships. Attempting to combine them raises a TypeError:

# This raises TypeError: can only concatenate str (not "bytes") to str
result = "Version: " + b"3.10"

---

2. History and Evolution of Character Sets

To understand encoding in Python, it helps to review how computers represent human language numerically.

+-------------------------------------------------------------------------+
|                                 UNICODE                                 |
|                   (U+0000 to U+10FFFF: 1,114,112 points)                |
|  +-------------------------------------------------------------------+  |
|  |                     ISO-8859-1 (Latin-1)                          |  |
|  |                       (0 to 255: 8-bit)                           |  |
|  |  +-------------------------------------------------------------+  |  |
|  |  |                         ASCII                               |  |  |
|  |  |                   (0 to 127: 7-bit)                         |  |  |
|  |  +-------------------------------------------------------------+  |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

ASCII (1963)

• Structure: 7-bit encoding defining 128 characters (code values 0 through 127).
• Contents: English uppercase letters (65–90 for 'A'–'Z'), lowercase letters (97–122 for 'a'–'z'), decimal digits (48–57 for '0'–'9'), punctuation marks, and non-printable control codes (10 for newline \n, 9 for tab \t, 0 for null).

Latin-1 / ISO-8859-1 (1987)

• Structure: 8-bit encoding defining 256 characters (0 through 255).
• Contents: The first 128 positions are identical to ASCII, while positions 128 through 255 add accented characters, umlauts, and currency symbols common in Western European languages (e.g., 'é', 'ñ', 'ü').

The Unicode Standard

Unicode assigns every character across all human languages, historical scripts, and symbol sets a unique numerical identifier called a code point. Unicode code points are conventionally written in hexadecimal notation with the prefix U+ (for example, U+0041 for 'A', U+03C0 for the Greek letter 'π', and U+1F600 for the emoji '😀'). Unicode defines 1,114,112 code points (U+0000 to U+10FFFF).

Unicode Transformation Formats (UTF)

Unicode defines the abstract code points, but an encoding determines how those numbers are translated into physical bytes:

---

3. Built-in Code Point Functions: ord() and chr()

Python provides two essential built-in functions for converting between characters and their numerical code points:

# ord(c): Character -> Integer Code Point
print(ord('A'))       # 65 (ASCII / Unicode)
print(ord('a'))       # 97
print(ord('0'))       # 48
print(ord('€'))       # 8364 (Euro sign)
print(ord('π'))       # 960 (Greek small letter pi)

# chr(i): Integer Code Point -> Character
print(chr(65))        # 'A'
print(chr(97))        # 'a'
print(chr(8364))      # '€'
print(chr(960))       # 'π'

Inverse Invariant Property

For any valid single-character string c and integer code point i: $\text{chr}(\text{ord}(c)) == c$ $\text{ord}(\text{chr}(i)) == i$

Exception Traps

• ord() accepts only a single character string (length exactly 1). Passing an empty string or multiple characters raises a TypeError:

  ord("")    # TypeError: ord() expected a character, but string of length 0 found
  ord("AB")  # TypeError: ord() expected a character, but string of length 2 found
  

• chr() accepts integers in the range 0 through 0x10FFFF (1,114,111). Passing a value outside this range raises a ValueError:

  chr(-1)          # ValueError: chr() arg not in range(0x110000)
  chr(1_114_112)   # ValueError: chr() arg not in range(0x110000)
  

---

4. String Encoding and Decoding Operations

Converting between str (characters) and bytes (octets) requires explicit encoding and decoding:

        str.encode(encoding="utf-8")
  str  =============================>  bytes
(Text) <============================= (Binary)
        bytes.decode(encoding="utf-8")

# 1. Encoding: str -> bytes
message = "Café"
utf8_bytes = message.encode("utf-8")
latin1_bytes = message.encode("latin-1")

print(utf8_bytes)    # b'Caf\xc3\xa9' (5 bytes: 'C', 'a', 'f' are 1 byte each; 'é' is 2 bytes)
print(latin1_bytes)  # b'Caf\xe9'     (4 bytes: 'é' is 1 byte in Latin-1)
print(len(utf8_bytes))    # 5
print(len(latin1_bytes))  # 4

# 2. Decoding: bytes -> str
recovered_text = utf8_bytes.decode("utf-8")
print(recovered_text)     # "Café"

Error Handling Strategies

When encoding or decoding encounters invalid or unrepresentable characters, the errors parameter dictates behavior:

• 'strict' (default): Raises UnicodeEncodeError or UnicodeDecodeError.
• 'ignore': Drops unencodable characters or undecodable bytes.
• 'replace': Inserts '?' during encoding, or the Unicode replacement character \ufffd ('') during decoding.

symbol = "pi: π"

# ASCII cannot represent 'π'
# strict mode (default)
# symbol.encode("ascii") -> UnicodeEncodeError

print(symbol.encode("ascii", errors="ignore"))   # b'pi: '
print(symbol.encode("ascii", errors="replace"))  # b'pi: ?'

---

5. String Immutability and the Python Memory Model

In Python, strings are immutable sequence types. Once a str object is allocated in memory, its contents can never be modified in-place.

language = "Python"

# Attempting in-place modification raises TypeError
try:
    language[0] = "J"
except TypeError as err:
    print(f"Error: {err}")
    # Output: Error: 'str' object does not support item assignment

Variable Rebinding vs. In-Place Mutation

When you perform operations like concatenation or calling string transformation methods, Python allocates a brand-new string object in memory and rebinds the variable name to point to the new address:

text = "Hello"
print(id(text))    # Memory address, e.g., 1402384920384

text += " World"   # Augmented assignment
print(id(text))    # Different memory address! A new str object was allocated

Performance Consideration: In performance-critical loops, repeatedly concatenating strings using += creates $O(N^2)$ time complexity due to repeated memory allocations and copying. The idiomatic Python approach is accumulating substrings in a list and using str.join().

String Interning

CPython optimizes memory usage by interning (caching in a lookup table) short strings, ASCII identifiers, and string literals. For interned strings, a is b evaluates to True because both references point to the exact same memory address. However, you should always use == for value equality comparisons, as dynamic runtime strings may not be interned.

---

6. Escape Characters and Raw Strings

Escape sequences allow you to embed non-printable control codes, quotes, and arbitrary Unicode characters inside string literals using the backslash \ character.

Common Escape Sequences

Raw String Literals (r"...")

Prefixing a string literal with r or R creates a raw string. In a raw string, backslashes are treated as literal characters rather than escape triggers:

# Standard string: \n is converted to a newline (length is 1)
normal_str = "\n"
print(len(normal_str))     # 1

# Raw string: \ and n are two separate literal characters (length is 2)
raw_str = r"\n"
print(len(raw_str))        # 2
print(raw_str)             # \n

# Windows file path use case
path = r"C:\new_folder\test.py"
print(path)                # C:\new_folder\test.py

Exam Trap: A raw string literal cannot end with an odd number of backslashes (e.g., r"C:\" causes a SyntaxError), because the trailing backslash escapes the closing quote.