5.7 Packet Analysis and Secure Protocols for Defenders

Key Takeaways

  • Wireshark and tcpdump capture and dissect packets; defenders use display filters (e.g., tcp.flags.syn==1, http.request) to triage floods, scans, and exfiltration.
  • Cleartext protocols (Telnet/23, FTP/21, HTTP/80, SNMPv1/2, POP3/110, IMAP/143) expose credentials to sniffing; their encrypted replacements are SSH/22, FTPS/SFTP, HTTPS/443, SNMPv3, and POP3S/IMAPS.
  • TLS provides confidentiality, integrity, and authentication; IPsec (AH for integrity, ESP for encryption) secures traffic at the network layer for VPNs.
  • Attack signatures are visible in captures: many SYNs without ACKs (flood/scan), gratuitous ARP replies (ARP poisoning), and oversized/abnormal DNS replies (amplification/tunneling).
  • Defenders enforce encryption everywhere, monitor with span ports/SIEM, and disable insecure legacy protocols to deny attackers the cleartext they rely on.
Last updated: June 2026

Packet Capture and Analysis Tools

Defenders read attacks directly off the wire with packet analyzers. Wireshark is the dominant GUI tool: it captures live traffic (in promiscuous or monitor mode) and dissects every protocol layer, while tcpdump is its command-line counterpart for headless capture and scripting. Captures are stored in pcap/pcapng files for offline analysis or sharing with a SIEM.

The key skill is filtering. Wireshark distinguishes capture filters (BPF syntax, applied before capture, e.g. tcp port 80) from display filters (applied to an existing capture, e.g. http.request). Useful defensive display filters:

Display FilterSurfaces
tcp.flags.syn==1 && tcp.flags.ack==0SYN floods / port scans
http.requestWeb requests (and Layer-7 floods)
dnsDNS queries (amplification/tunneling)
arp.duplicate-address-detectedARP poisoning / MITM
icmpPing sweeps, ICMP tunneling
tcp.analysis.retransmissionNetwork stress / desync

Reading the TCP flags, sequence numbers, and timing lets an analyst confirm whether traffic is a handshake, a flood, a hijack (the ACK storm of §5.2), or a scan.

Reading Attack Signatures in Captures

Many attacks in this chapter leave distinctive fingerprints a defender can recognize in Wireshark:

  • SYN flood / scan — a torrent of SYN packets with no corresponding ACK completing the handshake; many half-open SYN_RECV entries.
  • Smurf / amplification — a flood of ICMP echo replies or large UDP responses (DNS/NTP) arriving unsolicited at the victim.
  • ARP poisoning (MITM) — unsolicited or duplicate gratuitous ARP replies mapping the gateway IP to the attacker's MAC; Wireshark flags duplicate-address-detected.
  • DNS tunneling — abnormally long, high-entropy, or high-volume DNS queries to one domain (TXT/NULL records carrying encoded data).
  • Session hijacking — the ACK storm, out-of-sequence segments, and an injected RST.
  • Cleartext credential theftFollow TCP Stream on a Telnet/FTP/HTTP session reveals usernames and passwords in plaintext.

This is precisely why ethical hackers demonstrate sniffing: to prove that unencrypted protocols hand credentials to anyone on the segment, justifying the move to encryption.

Cleartext vs. Secure Protocols

The single highest-impact defensive measure against sniffing and hijacking is to eliminate cleartext protocols. CEH expects you to pair each insecure protocol with its encrypted replacement and remember the ports:

Insecure (Cleartext)PortSecure ReplacementPort
Telnet23SSH22
FTP21/20SFTP / FTPS22 / 990
HTTP80HTTPS (TLS)443
SNMP v1/v2c161SNMPv3 (auth+priv)161
POP3 / IMAP110 / 143POP3S / IMAPS995 / 993
SMTP (open)25SMTPS / STARTTLS465 / 587
LDAP389LDAPS636
rlogin/rsh/rcp513/514SSH22

Legacy protocols transmit credentials and data in the clear, so a single sniffer (Wireshark, Ettercap) on the segment harvests everything. Their encrypted replacements wrap the same function in TLS or SSH, defeating both passive sniffing and active injection.

TLS, IPsec, and Defensive Practice

3** removes weak ciphers and speeds the handshake. IPsec secures traffic at the network layer, transparently to applications, and is the backbone of site-to-site and remote-access VPNs. Know its two protocols: AH (Authentication Header) provides integrity and authentication but no encryption, while ESP (Encapsulating Security Payload) provides encryption plus integrity — ESP is what actually keeps payloads confidential. IPsec runs in transport mode (protects payload, host-to-host) or tunnel mode (protects the whole packet, gateway-to-gateway VPN).

Defensive practice for a network analyst:

  • Encrypt everything in transit — TLS for apps, SSH for admin, IPsec/VPN for site links.
  • Disable legacy/cleartext services and weak ciphers (SSLv3, TLS 1.0/1.1, SNMPv1).
  • Capture at choke points via SPAN/TAP and feed pcap/NetFlow to the SIEM for correlation.
  • Inspect TLS at the perimeter (NGFW decryption) so encryption does not blind the IDS.
  • Baseline normal traffic so floods, beaconing, and tunneling stand out.

Two caveats CEH expects you to hold together. First, encryption does not make traffic invisible — metadata (who talks to whom, when, how much) still leaks via NetFlow, and TLS inspection at the perimeter is needed so that encryption does not become an attacker's cover. Second, switched networks do not automatically prevent sniffing: an attacker on the LAN can use ARP poisoning or MAC flooding (forcing the switch to fail-open like a hub) to capture traffic that the switch would normally isolate — which is exactly why encryption, not network topology, is the real protection.

Port security, DHCP snooping, and Dynamic ARP Inspection harden the switch against those sniffing setups.

The exam synthesis: packet analysis lets defenders see the attacks taught in this chapter, and replacing cleartext protocols with TLS/SSH/IPsec removes the sniffable, injectable traffic that sniffing, MITM, and session hijacking all depend on. In short, the analyst's two jobs are to see (capture and filter intelligently) and to deny visibility to attackers (encrypt everything and disable legacy cleartext services).

For the exam, anchor the cleartext-to-secure protocol swaps cold: Telnet (23) → SSH (22), FTP (21) → SFTP/FTPS, HTTP (80) → HTTPS (443), POP3 (110)/IMAP (143) → POP3S (995)/IMAPS (993), SNMPv1/v2 → SNMPv3, and LDAP (389) → LDAPS (636). The lesson the CEH wants is that confidentiality comes from encryption, not obscurity: a sniffer on a switched network using ARP poisoning can still read any protocol that transmits credentials in plaintext, so encrypting the channel (TLS, SSH, IPsec) is the durable countermeasure rather than relying on the switch fabric alone.

Test Your Knowledge

A defender filters a Wireshark capture with tcp.flags.syn==1 && tcp.flags.ack==0 and sees thousands of such packets with no completing ACKs. Which activity is this most consistent with?

A
B
C
D
Test Your Knowledge

Within IPsec, which protocol provides encryption (confidentiality) of the payload, as opposed to integrity/authentication only?

A
B
C
D
Test Your Knowledge

An administrator still manages switches over Telnet (port 23). What is the best security improvement and why?

A
B
C
D