12.1 Understand and Implement Secure Protocols

Understand and implement secure protocols

Domain 5 of the ISC2 Systems Security Certified Practitioner (SSCP) exam — Cryptography, weighted at 9% under the outline effective 1 October 2025 — includes Knowledge Area 5.3: understand and implement secure protocols. Chapter 11 covered why you encrypt and which primitives you pick. This section is the operations job: put the right protocol on the right port for the right use case, and know what still leaks if you pick the ancestor protocol.

The outline names three piles: services and protocols, common use cases (credit card processing, file transfer, web client, virtual private network (VPN), transmission of personally identifiable information (PII)), and limitations and vulnerabilities. Translations of the outline explicitly list Internet Protocol Security (IPsec), Transport Layer Security (TLS), Secure/Multipurpose Internet Mail Extensions (S/MIME), and DomainKeys Identified Mail (DKIM). Expect items that look like a Tuesday change window, not a cipher-suite trivia contest.

A protocol is the agreed sequence and encoding two systems use so cryptography actually happens. A service is the job the user wanted — browse, copy a file, open a shell, send mail, join a remote network. SSCP fails you when you confuse a service with its secure wrapping. "We have file transfer" is not an answer. File Transfer Protocol (FTP) on port 21 sends credentials and payload in the clear. SSH File Transfer Protocol (SFTP) and FTP over TLS (FTPS) are the replacements, and they are not two names for the same thing.

Protocol replacements and ports

Memorize the insecure ancestor, the secure replacement, and the port. Domain 6 will reuse these ports in the networking catalog; Domain 5 asks whether the crypto is actually on.

If an item shows a scanner finding "TCP/21 open, anonymous login, payroll.csv," the fix is not "turn on Advanced Encryption Standard (AES) in the FTP banner." Replace the service.

Use cases the outline actually names

TLS and the web client

TLS is the session protocol that replaced Secure Sockets Layer (SSL). SSLv2 and SSLv3 are retired. TLS 1.0 and TLS 1.1 are deprecated by the Internet Engineering Task Force and rejected by current browsers and PCI Data Security Standard (PCI DSS) strong-cryptography expectations. For SSCP operations, TLS 1.2 is the minimum you still see in enterprises; TLS 1.3 is the preferred default (one round trip, no static RSA key-transport, broken ciphers removed).

HTTPS is HTTP carried inside a TLS session, almost always on port 443. It is not a separate application protocol with a different verb set. The browser (or API client) still speaks HTTP; TLS supplies confidentiality, integrity, and server authentication via a certificate that Knowledge Area 5.4 will make you issue and revoke. Mutual TLS (mTLS) is the same handshake with the client also presenting a certificate — common for service-to-service APIs, not for a public retail homepage.

Conceptually, the TLS handshake is an operations sequence, not a packet-trace hobby: the client offers versions and cipher suites; the server picks a suite and presents its certificate; both sides complete a key agreement; application data (the HTTP GET, the payment POST) then rides the record layer. Your job is to disable the suites and versions that still negotiate, to install a certificate whose names match the site, and to fail closed when validation fails. A user who clicks through a name-mismatch warning has just opted into a man-in-the-middle (MITM) path. A load balancer that terminates TLS 1.0 "for the old inventory scanner" has reopened a PCI and PII hole on the same listener that serves checkout.

Credit card processing. PCI DSS requires strong cryptography when cardholder data crosses open, public networks. A marketing homepage on HTTPS with a checkout form that still POSTs to http://pay.example.com on port 80 is a Domain 5.3 miss. The PAN, expiration, and security code leave the browser in the clear. Hashing the PAN in the page does not create a TLS session, does not authenticate the payment host, and is not a compensating control for transit. Redirect HTTP to HTTPS, turn on HSTS, disable TLS 1.0/1.1, and keep PAN out of web-server logs. Domain 5.1 already told you why PCI cares; 5.3 asks whether the checkout actually speaks TLS.

Scenario — PCI web checkout. Priya's retail site shows a padlock on the catalog. The payment plugin, last upgraded in 2019, posts the card form to an HTTP endpoint "because the processor's sample code used port 80." A packet capture on the guest Wi-Fi reads card numbers. The SSCP fix is not a new hash in Chapter 11. Terminate TLS on the payment host, move the post to HTTPS, and treat remaining HTTP as a redirect only — never as a place cardholder data is accepted.

File transfer: FTP, FTPS, and SFTP

Three names, two secure protocols, one museum piece.

• FTP (port 21): control channel in the clear, data channel in the clear, passwords in the clear. Active mode also fights firewalls. It is not acceptable for PII, protected health information (PHI), PAN, or source code.
• FTPS: FTP over TLS. Explicit FTPS upgrades port 21 with the AUTH TLS command. Implicit FTPS traditionally uses port 990. You still have FTP's two-channel personality, so firewalls must allow the TLS data ports. The payload and credentials are protected if TLS is configured to modern versions.
• SFTP: a different protocol — a subsystem of SSH on port 22. One channel, one port, SSH user authentication (keys preferred). It is not "FTP tunneled through HTTPS," and it is not FTPS with a typo.

Secure Copy (SCP) also rides SSH. Prefer SFTP for scripted vendor drops; it has directory listings and better resume behavior. If a vendor insists on "FTP" in the contract, read the next sentence: they often mean SFTP and have never distinguished the acronyms. Confirm the port. Port 22 is SFTP/SSH. Port 21 with a certificate is FTPS. Port 21 with no TLS is a finding.

Scenario — PII file transfer. HR drops new_hires.csv (names, Social Security numbers, start dates) on ftp://files.payroll-vendor.example with a shared password taped to the runbook. Encryption at rest on the vendor's disk does not protect the transfer. Move the drop to SFTP with a vendor-specific key, or FTPS if the vendor's stack is still FTP-shaped. Log who uploaded, and stop using a shared password as the identity.

SSH as the admin protocol

SSH replaced Telnet for remote shells, and it is also the transport under SFTP, SCP, and local port forwards. Default port 22. Authenticate with public keys; disable password authentication on internet-facing listeners; disable SSH protocol 1. Host-key verification (trust on first use, then pin) is how you stop a MITM on the jump host. SSH-agent forwarding and jumping through bastion hosts are operational conveniences that expand the blast radius of a stolen laptop — treat them as access decisions, not defaults. Domain 6 will place SSH among remote-access controls; here, remember it is the crypto protocol that makes a shell confidential and authenticated.

IPsec and the VPN use case

IPsec protects IP packets themselves (OSI layer 3), which is why it can wrap any inner protocol: file copies, remote desktop, LDAP, even HTTP that was never upgraded. Two payload protocols matter on the exam:

Tunnel mode encrypts the original IP packet and adds a new outer header — the usual site-to-site VPN. Transport mode encrypts the payload and leaves the original IP header — host-to-host. IKE (UDP 500, and UDP 4500 when NAT traversal is on) negotiates security associations. Peers authenticate with a pre-shared key (PSK) or with certificates. PSKs that are reused across every branch, stored in a wiki, and never rotated are a 5.3 limitation, not a 5.2 math problem. Certificate-authenticated IPsec fails closed when the cert is expired, revoked, or name-mismatched — which is correct cryptographically and catastrophic operationally if nobody monitors notAfter.

Scenario — expired cert takes down the VPN. Marcus runs IPsec between headquarters and the warehouse. Both gateways use machine certificates from the internal certificate authority (CA). The warehouse certificate expires Saturday. IKE cannot validate the peer. Inventory scanners cannot upload, even though the company's public HTTPS site is healthy on a different certificate. The limitation is not "IPsec is weak." The limitation is that implementing IPsec includes certificate lifetime, revocation checking, and an alert before expiry. Swap in a renewed cert, then add monitoring so the next one does not page you from a dark warehouse.

Split-tunnel VPNs send only selected prefixes through IPsec and let the rest of the laptop's traffic hit the internet directly. That is an availability and performance choice that can leak PII if a payroll session is not in the encrypted selector list. Full-tunnel sends everything through the gateway. SSCP items will ask which traffic is actually protected, not which vendor checkbox is prettier.

A TLS-based VPN (many client portals on 443) is a valid VPN implementation of 5.3. PPTP is not. If the stem still offers PPTP as the "easy remote-access option," treat it as a vulnerability.

Mail: S/MIME versus DKIM

Both appear in outline translations. They solve different problems.

S/MIME uses X.509 certificates (the same PKI idea as TLS, different extended key usage) to sign and/or encrypt a message. Encryption is end-to-end if both mail clients hold keys and the gateways do not strip the envelope. To encrypt, you need the recipient's public certificate. S/MIME is how you protect a settlement PDF in an email body so hops cannot read it. The operational costs are certificate distribution, mobile-mail support, and the fight with mail-gateway data loss prevention (DLP) that cannot scan ciphertext.

DKIM is a domain signature. The sending system hashes selected headers and the body, signs with a domain private key, and publishes the public key in Domain Name System (DNS) as a TXT record. Receivers verify that this domain authorized the message. DKIM does not encrypt. Combined with Sender Policy Framework (SPF) and Domain-based Message Authentication, Reporting, and Conformance (DMARC), it is an authenticity control against spoofing. It is the wrong tool when legal asked for "encrypted email."

Scenario. Counsel wants a social-security extract sent "securely" to outside counsel. IT enables DKIM on the corporate domain and declares mail secure. Every relay can still read the extract. The SSCP answer is S/MIME (if both sides have certificates) or, more often in operations, an HTTPS portal with TLS and access control — because bulk PII does not belong in a mailbox at all.

Limitations and vulnerabilities you will be asked to name

• Deprecated versions and ciphers: SSLv3, TLS 1.0/1.1, RC4, export suites, Telnet, FTP, PPTP. Leaving them enabled for "one old scanner" reopens the use case you thought you closed.
• Certificate problems: expiry, wrong name, missing intermediate, users bypassing warnings. Expiry is an availability outage (VPN, API, checkout) as often as it is a crypto failure.
• Downgrade and MITM: a network that can strip STARTTLS or offer HTTP will, unless you require TLS and pin or use HSTS.
• Inspection versus confidentiality: enterprise TLS intercept (a corporate root in the trust store) lets DLP see PII and also means you no longer have end-to-end TLS to the real server. Know which you configured.
• Wrong protocol for the goal: DKIM is not confidentiality. AH is not confidentiality. Hashing in the browser is not TLS. FTP after landing-disk encryption is not transit protection.
• Key and credential reuse: SSH keys copied to every jump box, IPsec PSKs in a shared chat, FTPS passwords in a batch file.
• Split tunnel and mixed content: the session you cared about never entered the protected protocol.

When you sit the exam, ask: what service is the human trying to do, which protocol actually wraps it, and what still leaks if that wrap is missing, expired, or the wrong tool? Knowledge Area 5.4 will then ask who issued the certificate that made TLS and IPsec and S/MIME possible.

Section takeaways

Put TLS on web and payment traffic (HTTPS 443), replace FTP with SFTP or FTPS, replace Telnet with SSH, and use IPsec ESP or a TLS VPN for remote networks. Use S/MIME when mail itself must be confidential; use DKIM when the domain must be authentic. Treat expired certificates, TLS 1.0, and cleartext ancestors as implementation failures of 5.3, not as trivia from a different domain.