2.2 TCP and UDP Transport Protocols & Handshake Mechanics

2.2 TCP and UDP Transport Protocols & Handshake Mechanics

At Layer 4 (Transport Layer) of the OSI model, protocols facilitate end-to-end communication services for applications. While Layer 3 (IP) routes datagrams from host to host, Layer 4 is responsible for process-to-process delivery through port multiplexing. For the CREST CPSA examination, candidates must master the internal mechanics, flag behaviors, state transitions, and diagnostic signatures of the Transmission Control Protocol (TCP) and User Datagram Protocol (UDP).

---

1. Transport Layer Port Architecture

Both TCP and UDP utilize 16-bit port numbers (values 0 to 65,535) to identify specific application processes on an endpoint. The combination of an IP address and a port number defines a socket (e.g., 192.168.1.50:443).

Port ranges are divided by IANA into three distinct tiers:

1. Well-Known Ports (0 – 1023): Assigned to core system services and protocols (e.g., HTTP 80, HTTPS 443, SSH 22, DNS 53). On Unix-like systems, binding to a well-known port historically requires superuser (root) privileges.
2. Registered Ports (1024 – 49151): Assigned to user processes, third-party applications, and databases (e.g., Microsoft SQL Server 1433, MySQL 3306, RDP 3389).
3. Dynamic / Private / Ephemeral Ports (49152 – 65535): Automatically assigned by the operating system kernel to client-side outbound connections for the duration of a session.

---

2. Transmission Control Protocol (TCP) Architecture

Defined in RFC 793 (and updated by RFC 9293), TCP is a connection-oriented, reliable, full-duplex byte-stream protocol. Before data can be transmitted, a formal logical connection must be established between client and server.

Reliability Mechanics

• Ordered Delivery: TCP numbers every byte sent using 32-bit Sequence Numbers, allowing the receiver to reassemble out-of-order packets into a contiguous stream.
• Positive Acknowledgment with Retransmission (PAR): The receiver returns Acknowledgment Numbers confirming receipt. If the sender does not receive an acknowledgment before a Retransmission Timeout (RTO) expires, the data is retransmitted.
• Flow Control: TCP uses a Sliding Window mechanism where the receiving host advertises its available buffer space (Window Size) to prevent buffer overrun.

TCP Header Structure

The TCP header has a minimum length of 20 bytes and can expand up to 60 bytes when options are included:

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|          Source Port          |       Destination Port        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                        Sequence Number                        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Acknowledgment Number                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Data |       |C|E|U|A|P|R|S|F|                               |
| Offset| Rsrvd |W|C|R|C|S|S|Y|I|          Window Size          |
|       |       |R|E|G|K|H|T|N|N|                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           Checksum            |        Urgent Pointer         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Options                    |    Padding    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Field Breakdown & Flag Analysis

• Source Port & Destination Port (16 bits each): Sockets for sender and receiver.
• Sequence Number (32 bits): In a SYN packet, this represents the Initial Sequence Number (ISN). In regular data transfer, it represents the sequence number of the first data byte in the segment.
• Acknowledgment Number (32 bits): If the ACK flag is set, this field contains the next sequence number the receiver expects to receive. Acknowledgment is cumulative ($Ack = Seq_{received} + Length_{data} + 1$).
• Data Offset (4 bits): Header length expressed in 32-bit (4-byte) words. Minimum value is 5 ($5 \times 4 = 20$ bytes); maximum is 15 ($15 \times 4 = 60$ bytes).
• Reserved (3 bits): Reserved for future use (must be 0).
• Control Flags (9 bits):
  • CWR (Congestion Window Reduced): Sender reduced its transmission rate (RFC 3168).
  • ECE (ECN-Echo): Indicates network congestion encountered (RFC 3168).
  • URG (Urgent): Indicates the Urgent Pointer field is significant.
  • ACK (Acknowledgment): Indicates the Acknowledgment Number field is valid. Once established, all TCP segments carry ACK.
  • PSH (Push): Instructs the receiver to push buffered data immediately up to the application layer rather than waiting for buffers to fill.
  • RST (Reset): Aborts an existing connection or rejects an incoming connection request to a closed port.
  • SYN (Synchronize): Synchronizes sequence numbers to initiate a connection.
  • FIN (Finish): Signals that the sender has finished transmitting data and requests graceful teardown.
  • NS (Nonce Sum): Protects against accidental or malicious concealment of marked packets (RFC 3540).
• Window Size (16 bits): Number of data octets the sender is willing to accept without further acknowledgment.
• Checksum (16 bits): Mandatory error-checking field covering the TCP header, TCP payload, and a 12-byte IP Pseudo-Header (Source IP, Destination IP, Reserved, Protocol 6, TCP Segment Length).
• Urgent Pointer (16 bits): Byte offset from the sequence number pointing to the end of urgent data.
• TCP Options (Variable): Common options negotiated during the handshake:
  • Maximum Segment Size (MSS, Kind 2): Maximum data payload a host can receive in a single segment (typically 1460 bytes for a 1500-byte Ethernet MTU).
  • Window Scale (Kind 3, RFC 7323): Multiplier allowing the 16-bit Window field to scale up to $1\text{ GB}$ for high-bandwidth/high-latency networks.
  • Selective Acknowledgment (SACK Permitted, Kind 4, RFC 2018): Enables receiver to acknowledge discontinuous blocks of received data.
  • Timestamps (Kind 8, RFC 7323): Used for Round Trip Time Measurement (RTTM) and Protection Against Wrapped Sequences (PAWS). Can leak remote system uptime during reconnaissance.

Initial Sequence Number (ISN) Generation & Security

In early TCP/IP implementations, the ISN was incremented predictably using a fixed counter. In 1995, Kevin Mitnick famously exploited this via TCP Blind Spoofing / Sequence Number Prediction: by flooding a trusted server with SYN packets to silence it, an attacker could spoof the trusted server's IP address against a target host, blindly predict the target's ISN response, and transmit a forged ACK to establish a one-way authenticated rsh/rlogin session.

Modern operating systems implement RFC 1948 and cryptographically secure pseudorandom number generators (PRNGs) to randomize ISNs, preventing blind sequence number prediction.

---

3. TCP Handshake Mechanics & State Machine

   Client (Active Open)                             Server (Passive Open)
      [CLOSED]                                            [LISTEN]
         |                                                   |
         | -------------- 1. SYN (Seq=x) ------------------> |  Server creates TCB
    [SYN_SENT]                                          [SYN_RECEIVED]
         |                                                   |
         | <--------- 2. SYN-ACK (Seq=y, Ack=x+1) ---------- |
    [ESTABLISHED]                                            |
         |                                                   |
         | ------------- 3. ACK (Seq=x+1, Ack=y+1) --------> |
         |                                              [ESTABLISHED]
         | <=========== Full-Duplex Data Flow =============> |

The 3-Way Handshake Step-by-Step

1. SYN (Step 1): Client sends a segment with SYN=1, a randomly generated client sequence number Seq = x, and proposed TCP Options (MSS, SACK, Window Scale). Client enters SYN_SENT.
2. SYN-ACK (Step 2): Server allocates a Transmission Control Block (TCB) memory buffer, selects its own random sequence number Seq = y, acknowledges the client's sequence number with Ack = x + 1, and sets SYN=1, ACK=1. Server enters SYN_RECEIVED.
3. ACK (Step 3): Client sends ACK=1 with Seq = x + 1 and Ack = y + 1. Both endpoints enter ESTABLISHED.

SYN Flood Attack & SYN Cookies

Because the server allocates memory for the TCB upon receiving Step 1 (SYN) before the client confirms Step 3 (ACK), an attacker can flood thousands of SYN packets with spoofed source IPs. The server fills its connection backlog queue with half-open connections in the SYN_RECEIVED state, exhausting system memory and causing a Denial of Service (DoS).

• Defense - SYN Cookies (RFC 4987): When the backlog queue fills, the server stops allocating TCB memory. Instead, it encodes the connection state, MSS, and timestamp cryptographically into the server's ISN ($y$). Only when the client returns a valid ACK ($y + 1$) does the server reconstruct the TCB from the acknowledgment number.

Connection Teardown: 4-Way Handshake & RST Teardown

TCP connections terminate via a graceful 4-way handshake because TCP is full-duplex and each direction must close independently:

   Active Closer                                       Passive Closer
    [ESTABLISHED]                                       [ESTABLISHED]
         |                                                   |
         | --------------- 1. FIN (Seq=u) -----------------> |  Enters [CLOSE_WAIT]
    [FIN_WAIT_1]                                             |
         | <-------------- 2. ACK (Ack=u+1) ---------------- |  Application notified
    [FIN_WAIT_2]                                             |
         |                                                   |  Sends remaining data
         | <-------------- 3. FIN (Seq=v) ------------------ |  Enters [LAST_ACK]
         |                                                   |
         | --------------- 4. ACK (Ack=v+1) ---------------> |
    [TIME_WAIT]                                           [CLOSED]
         | (Wait 2 * MSL)
      [CLOSED]

• TIME_WAIT State: The active closer remains in TIME_WAIT for 2 Maximum Segment Lifetimes (2 * MSL), typically 60 to 120 seconds. This guarantees that the final ACK was delivered (if lost, the peer retransmits FIN) and ensures that lingering duplicate packets from this connection expire before the socket is reused.
• RST Abortive Teardown: A connection can be instantaneously aborted by transmitting a segment with RST=1. Any buffered in-flight data is discarded immediately without waiting for acknowledgments.

---

4. User Datagram Protocol (UDP) Architecture

Defined in RFC 768, UDP is a connectionless, lightweight, unreliable transport protocol. It operates on a best-effort delivery model with no handshakes, no acknowledgments, no sequence ordering, and no flow or congestion control.

Fixed 8-Byte Header Structure

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|          Source Port          |       Destination Port        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|            Length             |           Checksum            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                            Payload                            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

1. Source Port (16 bits): Identifies sending process; optional in IPv4 (set to 0 if unused).
2. Destination Port (16 bits): Target application port.
3. Length (16 bits): Total length of UDP header and payload in bytes (minimum value is 8).
4. Checksum (16 bits): Covers UDP header, payload, and the IP pseudo-header. Optional in IPv4, but strictly mandatory in IPv6.

Common UDP Services Tested on CPSA

UDP Amplification & Reflection Attacks

Because UDP is connectionless and does not perform a handshake, an attacker can forge (spoof) the packet's Source IP address to match a victim's IP. The attacker transmits small requests to publicly exposed third-party UDP servers (reflectors) that generate massive responses directed at the victim. Amplification factors:

• DNS: 20x to 50x amplification using ANY or large EDNS0 queries.
• NTP: Over 500x amplification using the legacy monlist diagnostic command.

---

5. Transport Layer Port Scanning Mechanics

Understanding how transport layer stacks respond to unexpected or malformed probes is the foundation of network reconnaissance with tools like Nmap.

TCP Port Scanning Methodologies

1. TCP Connect Scan (nmap -sT):
   • Uses the OS connect() API to complete the full 3-way handshake (SYN -> SYN-ACK -> ACK).
   • Advantage: Does not require raw socket / root privileges.
   • Disadvantage: Easily detected; leaves explicit connection log entries in application event logs.
2. TCP SYN Scan / Stealth Scan (nmap -sS):
   • The scanner sends a SYN packet.
   • Open Port: Target responds with SYN-ACK. Scanner immediately returns a RST to tear down the connection before the handshake completes. The application layer never sees an established connection.
   • Closed Port: Target responds with RST-ACK.
   • Filtered Port: No response received (packet dropped by firewall) or target returns ICMP Type 3 error.
   • Advantage: Fast, stealthy; requires root/raw socket access.
3. TCP Inverse & Malformed Scans (RFC 793 Compliant Scans):
   • NULL Scan (-sN): No flags set (0x00).
   • FIN Scan (-sF): Only FIN set.
   • XMAS Scan (-sX): FIN, PSH, and URG set simultaneously.
   • Rule: According to RFC 793, if a closed port receives any packet lacking SYN, RST, or ACK, it must respond with a RST. If the port is open, the packet is silently discarded.
   • OS Fingerprint Limitation: Microsoft Windows, Cisco, and BSDi operating systems violate RFC 793 and send RST for all ports regardless of state, rendering inverse scans ineffective against Windows environments.

UDP Port Scanning Mechanics (nmap -sU)

Scanning UDP is notoriously slow and difficult due to the absence of handshakes:

• Probe: Scanner sends an empty or protocol-specific UDP datagram to the target port.
• Response Signatures:
  • If the target returns a UDP data response $\rightarrow$ Port is OPEN.
  • If the target returns an ICMP Type 3 Code 3 (Port Unreachable) message $\rightarrow$ Port is CLOSED.
  • If no response is received (or ICMP Type 3 Codes 1, 2, 9, 10, or 13) $\rightarrow$ Port is classified as OPEN|FILTERED (the packet could be open and silently accepted, or dropped by a firewall).
• Kernel Rate Limiting: Operating systems enforce RFC 1812 rate limiting on ICMP error generation (e.g., Linux limits ICMP Destination Unreachable responses to 1 per second). Consequently, a comprehensive 65,535-port UDP scan can require hours to complete.