3.3 Secure Protocols: SSL/TLS, IPsec & SSH

Secure Protocols: SSL/TLS, IPsec & SSH

Quick Summary: Applying cryptographic primitives across real-world computer networks requires structured communication protocols. In the CREST CPSA syllabus, secure protocols represent approximately 10% of technical evaluations. Candidates must demonstrate rigorous knowledge of Transport Layer Security (TLS 1.2 vs. TLS 1.3 handshake mechanics, legacy flaws, and historical CVEs), Internet Protocol Security (IPsec AH vs. ESP, Transport vs. Tunnel modes, and IKE Phase 1/2 negotiation), Secure Shell (SSHv2 architecture and port forwarding techniques), and Pretty Good Privacy (PGP/GPG hybrid encryption).

---

Transport Layer Security (SSL/TLS) Evolution & Deprecation

Transport Layer Security (TLS) and its predecessor, Secure Sockets Layer (SSL), operate above the transport layer (TCP) and beneath application layer protocols (HTTP, SMTP, IMAP) to provide end-to-end channel encryption, endpoint authentication, and message integrity.

Historical Evolution and Protocol Deprecation

• SSL 2.0 (1995): Severely flawed; vulnerable to cipher downgrade attacks, identical keys for message authentication and encryption, and insecure handshake truncation. Formally deprecated under RFC 6176.
• SSL 3.0 (1996): Introduced SHA-1 and MD5 MACs with standard PKI support, but fundamentally vulnerable to the POODLE CBC padding oracle attack. Formally deprecated under RFC 7568.
• TLS 1.0 (1999 / RFC 2246): Minor upgrade over SSL 3.0. Vulnerable to predictable CBC IV attacks (BEAST). Formally deprecated by IETF under RFC 8996 in 2021.
• TLS 1.1 (2006 / RFC 4346): Introduced explicit CBC IVs to remediate BEAST, but retained legacy cryptographic primitives (MD5/SHA-1 signatures). Formally deprecated under RFC 8996.
• TLS 1.2 (2008 / RFC 5246): Added authenticated encryption (AES-GCM), configurable PRF based on SHA-256, and modern extension parameters. Remains widely deployed, but legacy configurations permitting static RSA key exchange or CBC ciphers are considered insecure.
• TLS 1.3 (2018 / RFC 8446): Complete architectural overhaul designed for maximum security and minimal latency.

Architectural Contrast: TLS 1.2 vs. TLS 1.3

Why TLS 1.3 Mandates PFS and Eliminates Static RSA

In TLS 1.2 with static RSA key exchange (TLS_RSA_WITH_AES_128_CBC_SHA), the client generates a Pre-Master Secret, encrypts it with the server's static public key, and transmits it. If an adversary passively records this traffic and obtains the server's private key years later (e.g., via server decommission, subpoena, or breach), they can decrypt every recorded session.

TLS 1.3 completely removes static RSA key exchange. By mandating Ephemeral Diffie-Hellman (DHE or ECDHE), TLS 1.3 guarantees Perfect Forward Secrecy (PFS): temporary session keys are generated dynamically and discarded immediately, preventing retroactive bulk decryption.

---

TLS Handshake Mechanics: Step-by-Step Execution

Standard TLS 1.2 Handshake (2-RTT)

Client                                                  Server
  │                                                       │
  │─── 1. ClientHello (TLS 1.2, Ciphers, ClientRandom, SNI) ───►│
  │                                                       │
  │◄── 2. ServerHello (Selected Cipher, ServerRandom) ────│
  │◄── 3. Certificate (Server X.509 Leaf + Intermediates) ─│
  │◄── 4. ServerKeyExchange (ECDHE Params + Signature) ───│
  │◄── 5. ServerHelloDone ────────────────────────────────│
  │                                                       │
  │─── 6. ClientKeyExchange (Client ECDHE Public Param) ─►│
  │─── 7. ChangeCipherSpec ──────────────────────────────►│
  │─── 8. Finished (Encrypted Handshake Hash) ───────────►│
  │                                                       │
  │◄── 9. ChangeCipherSpec ───────────────────────────────│
  │◄── 10. Finished (Encrypted Handshake Hash) ───────────│
  │                                                       │
  │◄════════════ Secure Application Data (AES-GCM) ══════►│

1. ClientHello: Client transmits supported TLS versions, a list of supported cipher suites, a 32-byte ClientRandom, and TLS extensions, including Server Name Indication (SNI). SNI specifies the target hostname in cleartext, enabling virtual hosting of multiple SSL sites on a single IP address.
2. ServerHello: Server selects the highest mutual TLS version and strongest mutual cipher suite, returning its own 32-byte ServerRandom.
3. Certificate: Server transmits its X.509 certificate chain (leaf certificate and intermediate CAs).
4. ServerKeyExchange: If ephemeral key exchange (DHE/ECDHE) is negotiated, the server sends its ephemeral DH public parameter, cryptographically signed with the private key corresponding to its certificate.
5. ServerHelloDone: Server signals the completion of its negotiation stage.
6. ClientKeyExchange: Client validates the certificate chain against its local trust store, extracts the server's ephemeral DH parameter, and returns its own ephemeral DH public parameter.
7. Key Derivation: Both endpoints calculate the Pre-Master Secret from the DH exchange, then compute the 48-byte Master Secret using the Pseudorandom Function (PRF): $\text{Master Secret} = \text{PRF}(\text{Pre-Master Secret}, \text{"master secret"}, \text{ClientRandom} \parallel \text{ServerRandom})$ From the Master Secret, both sides derive symmetric session keys, IVs, and MAC keys.
8. ChangeCipherSpec & Finished: Both endpoints notify each other that subsequent traffic is encrypted, validating handshake integrity by exchanging an encrypted hash of all previous handshake messages.

TLS 1.3 Optimized Handshake (1-RTT)

In TLS 1.3, the client anticipates the server's choice by sending its cipher suites alongside its own ephemeral key shares directly inside the ClientHello. The server replies with its certificate, key share, and Finished message in a single round-trip, cutting handshake latency in half (1-RTT).

---

Historical SSL/TLS Vulnerabilities & Penetration Testing Triage

During infrastructure penetration tests, CPSA candidates regularly assess legacy TLS configurations. The following historical vulnerabilities must be understood conceptually:

1. Heartbleed (CVE-2014-0160)

• Vulnerability Class: Missing bounds check / Buffer over-read in OpenSSL 1.0.1 through 1.0.1f.
• Mechanism: Exploits the TLS Heartbeat Extension (RFC 6520), which allows keep-alive ping messages. A client sends a payload string and asserts its length (e.g., sending the 4-byte payload ping and asserting length 4). In vulnerable OpenSSL code, an attacker sends a 1-byte payload but falsely declares a payload length of 65,535 bytes (64 KB). The server allocates a 64 KB memory buffer, copies the 1-byte payload, and reads the remaining 65,534 bytes directly from adjacent process heap memory back to the attacker.
• Impact: Repeated requests leak unencrypted passwords, session tokens, and the server's private RSA encryption keys without leaving audit logs.

2. BEAST (Browser Exploit Against SSL/TLS / CVE-2011-3389)

• Vulnerability Class: Predictable Initialization Vectors in TLS 1.0 CBC mode.
• Mechanism: In TLS 1.0, the IV for block $N$ is the final ciphertext block of packet $N-1$ ($C_{N-1}$). An attacker with man-in-the-middle positioning and client-side JavaScript execution (injecting chosen plaintext) can predict the IV. By aligning target cookies on 8-byte block boundaries, the attacker decrypts session cookies byte-by-byte.
• Remediation: Disable TLS 1.0; upgrade to TLS 1.2/1.3.

3. POODLE (Padding Oracle On Downgraded Legacy Encryption / CVE-2014-3566)

• Vulnerability Class: Cryptographic padding oracle in SSL 3.0 CBC mode.
• Mechanism: SSL 3.0 does not specify the contents of padding bytes—it only verifies the final length byte. Active network attackers intentionally inject TCP RST packets to fail modern TLS handshakes, forcing browsers to downgrade to SSL 3.0. The attacker then shifts ciphertext blocks until a modified block matches a valid padding byte, decrypting session cookies in roughly 256 attempts per byte.
• Remediation: Completely disable SSL 3.0 on servers and clients.

4. CRIME (CVE-2012-4929) and BREACH (CVE-2013-3587)

• Vulnerability Class: Compression side-channel information leakage.
• Mechanism: Lossless data compression (DEFLATE) replaces repeated byte strings with short pointers. When an attacker injects guessed text into an HTTP request where sensitive secret tokens (e.g., authentication cookies in CRIME or CSRF tokens in BREACH) are reflected, matching characters compress more efficiently, producing smaller ciphertext packets. By observing packet size changes across encrypted channels, the attacker extracts secrets character-by-character.
• Remediation: Disable TLS-level compression (for CRIME) and disable HTTP body compression (GZIP/Brotli) on pages reflecting user input (for BREACH).

---

IPsec Architecture: Protocols, Modes & Key Management

Internet Protocol Security (IPsec) is an open-standard suite defined in RFC 4301 operating at the Network Layer (Layer 3) of the OSI model. It protects all protocols running above IP (TCP, UDP, ICMP, routing protocols).

IPsec Architecture:
1. Protocols ──────► AH (Protocol 51, Integrity/Auth ONLY) vs ESP (Protocol 50, Confidentiality + Integrity)
2. Modes ──────────► Transport Mode (Host-to-Host) vs Tunnel Mode (Gateway-to-Gateway / VPN)
3. Key Management ─► IKEv1/IKEv2 (UDP 500, NAT-T UDP 4500) ──► Phase 1 (ISAKMP SA) + Phase 2 (IPsec SA)

1. Security Protocols: AH vs. ESP

Authentication Header (AH / IP Protocol 51)

• Capabilities: Provides data integrity, data origin authentication, and optional anti-replay protection using HMAC (e.g., HMAC-SHA256). AH does NOT provide encryption/confidentiality.
• The NAT Incompatibility Problem: AH calculates its cryptographic Integrity Check Value (ICV) across the data payload AND the immutable fields of the IP header. When an IP packet crosses a Network Address Translation (NAT) router, the router rewrites the source or destination IP address. The receiving IPsec endpoint recalculates the ICV, detects the altered IP header, and drops the packet as corrupted or tampered with. AH is fundamentally incompatible with NAT.

Encapsulating Security Payload (ESP / IP Protocol 50)

• Capabilities: Provides confidentiality (symmetric encryption via AES-CBC or AES-GCM), data origin authentication, integrity, and anti-replay protection.
• NAT Compatibility: ESP calculates its ICV strictly across the ESP header, payload, and ESP trailer—excluding the outer IP header. To traverse stateful NAT devices that lack Protocol 50 translation, NAT-Traversal (NAT-T / RFC 3948) encapsulates the entire ESP packet inside a UDP header on UDP port 4500.

2. Modes of Operation: Transport Mode vs. Tunnel Mode

Original Packet:   [ IP Header ] [ TCP Header ] [ Data Payload ]

Transport Mode:    [ IP Header ] [ ESP Header ] [ TCP Header (Encrypted) ] [ Data (Encrypted) ] [ ESP Trailer ] [ ESP Auth ]
                   (Preserved)                  ◄──────────────────── Encrypted Scope ──────────────────────►

Tunnel Mode:       [ New Outer IP ] [ ESP Header ] [ IP Header (Orig) ] [ TCP Header ] [ Data ] [ ESP Trailer ] [ ESP Auth ]
                   (Gateway Addr)                  ◄──────────────── Encrypted Scope (Entire Packet) ──────────►

• Transport Mode (Host-to-Host): Encrypts and authenticates only the L4 transport payload (e.g., TCP segment). The original IP header is preserved and visible. Used for direct communication between two endpoints on the same internal network (e.g., server to management console).
• Tunnel Mode (Gateway-to-Gateway / Site-to-Site VPN): Encrypts the entire original IP packet (including the original source and destination IP addresses). It prepends an ESP header and a completely new outer IP header containing the gateway routers' public IP addresses. This completely masks the internal network addressing schema across transit networks.

3. Key Management: IKE Architecture (Phase 1 & Phase 2)

Internet Key Exchange (IKE / RFC 2409, RFC 7296) automates security negotiations over UDP port 500 (or UDP 4500 for NAT-T):

IKE Phase 1: Establish the IKE / ISAKMP Security Association

Establishes a secure, authenticated, bidirectional control channel (the ISAKMP SA) between the two IPsec endpoints. Can run in one of two modes:

• Main Mode (6 packets): Provides identity protection. Negotiates cryptographic parameters (packets 1-2), executes a Diffie-Hellman exchange (packets 3-4), and authenticates the endpoints (via pre-shared key or digital certificates) using encrypted identities (packets 5-6).
• Aggressive Mode (3 packets): Compresses negotiation into three packets for speed. In Aggressive Mode, the responder's identity, Diffie-Hellman parameters, and the pre-shared key (PSK) hash are transmitted in cleartext in the initial exchange.

CPSA Penetration Testing Finding: IKE Phase 1 Aggressive Mode with Pre-Shared Keys is a critical finding. An attacker can run ike-scan -A -M --id=groupname <target> to capture the PSK hash and crack it offline using hashcat or john without generating security logs on the firewall.

IKE Phase 2: Quick Mode (Establish IPsec SAs)

Protected inside the encrypted IKE Phase 1 management tunnel, Phase 2 negotiates the unidirectional IPsec Security Associations (SAs) for data traffic (two SAs required for bidirectional traffic: one inbound, one outbound). Quick Mode defines the symmetric cipher, hash algorithm, and traffic selectors (allowed subnets). It can optionally enforce Perfect Forward Secrecy (PFS) by executing a fresh Diffie-Hellman exchange.

---

Secure Shell (SSH) Protocol Architecture & Forwarding

Secure Shell (SSH / TCP port 22) provides encrypted remote administration, secure file transfer (SFTP/SCP), and network tunneling. It replaces insecure legacy cleartext protocols (Telnet, rlogin, rsh).

SSHv1 vs. SSHv2

• SSHv1: Architecturally broken. It used CRC32 for integrity, which is vulnerable to the CRC32 Compensation Attack (allowing an attacker to inject arbitrary packets into an active encrypted stream). SSHv1 lacked forward secrecy and is completely obsolete.
• SSHv2 (RFC 4251 - 4254): Fixed all structural flaws. Replaced CRC32 with HMAC/AEAD, separated transport and authentication layers, and mandated Diffie-Hellman key exchange with forward secrecy.

Three-Layer Architecture of SSHv2

1. Transport Layer (RFC 4253): Handles server host key authentication (verifying server authenticity), initial Diffie-Hellman key exchange, symmetric encryption, and integrity verification.
2. User Authentication Layer (RFC 4252): Authenticates the client to the server using passwords, public-key authentication (~/.ssh/authorized_keys), host-based authentication, or Kerberos/GSSAPI.
3. Connection Layer (RFC 4254): Multiplexes multiple logical communication channels across the single encrypted transport tunnel, providing interactive shells, command execution, and port forwarding.

SSH Port Forwarding & Pivoting Mechanics

For penetration testers and security analysts, SSH port forwarding is a primary technique for bypassing firewalls and pivoting into segmented networks.

SSH Port Forwarding Types:
1. Local (-L)   ──► Pulls remote service to local machine:       ssh -L 8080:internal.corp:80 user@jump
2. Remote (-R)  ──► Exposes local listener to remote server:      ssh -R 4444:localhost:4444 user@jump
3. Dynamic (-D) ──► Creates multi-port SOCKS proxy for pivoting:  ssh -D 1080 user@jump

1. Local Port Forwarding (ssh -L)

Binds a listening port on the local client machine. Traffic sent to this local port is forwarded across the encrypted SSH tunnel to the remote SSH server, which connects to the destination host on your behalf:

# Format: ssh -L [local_bind_ip:]<local_port>:<destination_ip>:<destination_port> user@jumphost
ssh -L 8080:192.168.10.50:80 user@jumphost.corp.com

Use Case: Accessing an internal web application (e.g., an admin console at 192.168.10.50:80) that is blocked by perimeter firewalls but reachable from jumphost.

2. Remote Port Forwarding (ssh -R)

Binds a listening port on the remote SSH server. Inbound connections arriving at that remote server port are tunneled backwards through the SSH connection to a destination accessible by the local client:

# Format: ssh -R [remote_bind_ip:]<remote_port>:<destination_ip>:<destination_port> user@jumphost
ssh -R 4444:localhost:4444 user@jumphost.corp.com

Use Case: Catching a reverse shell. An internal compromised host connects out to jumphost:4444, which forwards the incoming reverse shell directly back to the tester's local listener on localhost:4444.

3. Dynamic Port Forwarding (ssh -D)

Instantiates a local SOCKS4/SOCKS5 proxy server on the tester's machine. Rather than binding a single static destination port, dynamic port forwarding routes arbitrary TCP traffic to any internal destination on demand:

# Format: ssh -D <local_socks_port> user@jumphost
ssh -D 1080 user@jumphost.corp.com

Use Case: Dynamic network pivoting. By pairing this command with a tool like proxychains (configured to point to 127.0.0.1:1080), a penetration tester can run Nmap scans, web browsers, and exploitation tools across the target network through the SSH tunnel:

proxychains nmap -sT -Pn -p 80,443,445 192.168.10.0/24

---

Pretty Good Privacy (PGP) / GPG: Hybrid Cryptography

Pretty Good Privacy (PGP / RFC 4880 OpenPGP) and its open-source equivalent, GNU Privacy Guard (GPG), protect emails, documents, and code commits using a hybrid cryptosystem.

Why Hybrid Cryptosystems are Necessary

• Asymmetric encryption (RSA/ECC) is mathematically complex and computationally slow ($100\times$ to $1000\times$ slower than AES), making it impractical for encrypting large files.
• Symmetric encryption (AES) is fast, but suffers from the key distribution problem.

The Hybrid Encryption Lifecycle

Hybrid PGP Encryption Flow:
1. Large Plaintext Message ──► [ Encrypted with Random Session Key (AES-256) ] ──► Encrypted Payload
                                                        ▲
                                                        │
2. One-Time Random Symmetric Session Key ───────────────┴──► [ Encrypt with Recipient's RSA/ECC Public Key ]
                                                                           │
                                                                           ▼
3. Package Transmitted: [ Encrypted Session Key (Small) ]  +  [ Encrypted Payload (Large) ]

1. Encryption Phase:
   • Sender generates a cryptographically strong, random, one-time symmetric session key (e.g., AES-256).
   • The message payload is compressed and encrypted using this session key.
   • The session key is encrypted using the recipient's asymmetric public key.
   • The encrypted session key and encrypted payload are bundled together into a single transmission.
2. Decryption Phase:
   • Recipient uses their asymmetric private key to decrypt the symmetric session key.
   • Recipient uses the recovered session key to decrypt the large message payload.

Web of Trust (WoT) vs. PKI

Unlike enterprise TLS which relies on centralized, hierarchical Certificate Authorities, OpenPGP traditionally uses a decentralized Web of Trust (WoT). Individual users digitally sign each other's public keys after verifying identity, building a cumulative network of mutual trust assertions.