6.2 Port Scanning Mechanics: TCP, UDP & Stealth Scans

6.2 Port Scanning Mechanics: TCP, UDP & Stealth Scans

Port scanning is the methodical interrogation of transport layer port numbers (0 to 65,535 across TCP and UDP) on a target host to determine which network services are listening and accessible. In penetration testing, understanding the precise bit-level mechanics of each scan type is essential. Security analysts must accurately interpret scan output, navigate firewall filtering behaviors, and anticipate how target operating systems respond to malformed or unexpected transport segments.

---

Port State Definitions in Nmap

Nmap does not simply categorize a port as either "working" or "broken." Instead, it assigns each scanned port to one of six distinct port states based on the presence, absence, or type of response received:

1. open: An application daemon is actively accepting incoming TCP connections, UDP datagrams, or SCTP associations on this port. Identifying open ports is the primary objective of enumeration.
2. closed: The target host is reachable and processed the probe, but no application daemon is listening on the port. The target system actively responded with an explicit negative acknowledgment (a TCP RST packet or an ICMP Port Unreachable message).
3. filtered: An intermediate firewall, router access control list (ACL), or host-based firewall blocked the probe packet, preventing it from reaching the destination port, or blocked the target's response from returning to the scanner. Nmap cannot determine whether the port is open or closed because no response was received, or an explicit ICMP administrative filtering error was returned.
4. unfiltered: The port is accessible and reachable by the scanner's probes, but Nmap cannot determine whether an application is actively listening. This state is returned exclusively by scans such as the TCP ACK scan (-sA), which elicit identical RST replies from both open and closed ports.
5. open|filtered: Nmap cannot distinguish whether a port is open or filtered. This occurs when an open port provides no response (the expected behavior for UDP scans or inverse stealth scans) and packet filtering also results in no response.
6. closed|filtered: Nmap cannot determine whether a port is closed or filtered. This state appears only in specialized scans such as the IP ID Idle Scan (-sI).

---

TCP Port Scanning Mechanics: Full-Open vs Half-Open

The Transmission Control Protocol (TCP, RFC 793) is a connection-oriented protocol relying on a three-way handshake to establish reliable virtual circuits between endpoints:

Client (Scanner)                                  Server (Target)
      |                                                 |
      |------------------ 1. TCP SYN ------------------>| (Initiate Connection)
      |<----------------- 2. TCP SYN-ACK ---------------| (Acknowledge & Sync)
      |------------------ 3. TCP ACK ------------------>| (Handshake Complete)
      |                                                 |
      |<========== Established Data Channel ===========>|

1. TCP SYN Scan (-sS): The "Half-Open" Stealth Scan

The TCP SYN scan is Nmap's default and most widely utilized scan type when executed with administrative privileges (root/raw socket access). It is termed a half-open scan because it deliberately avoids completing the full three-way handshake.

                                TCP SYN SCAN (-sS)
[Port Open]
Scanner                                                       Target
   |----------------- 1. TCP SYN (Port 80) ------------------>|
   |<---------------- 2. TCP SYN-ACK -------------------------| (Port is Listening)
   |----------------- 3. TCP RST ---------------------------->| (Aborted by Scanner!)
   
[Port Closed]
Scanner                                                       Target
   |----------------- 1. TCP SYN (Port 81) ------------------>|
   |<---------------- 2. TCP RST -----------------------------| (No Service Listening)
   
[Port Filtered]
Scanner                                                       Target
   |----------------- 1. TCP SYN (Port 82) ------------------>|
   |                  [Dropped by Firewall]                   | (No Response / Timeout)
   |                           OR                             |
   |<---- ICMP Type 3 Code 1, 2, 3, 9, 10, or 13 ------------| (Filtered by Filter)

Execution Steps and Packet Flow:

1. The scanner generates a raw TCP packet with the SYN flag set targeting a specific destination port.
2. Open Response: If the port is open, the target TCP stack allocates transmission control blocks (TCB) and returns a SYN-ACK packet.
3. Tear-Down: As soon as the scanner's raw socket engine receives the SYN-ACK, it immediately transmits a RST packet back to the target. The connection is torn down instantly without sending the final ACK.
4. Closed Response: If the port is closed, the target TCP stack immediately returns a RST packet.
5. Filtered Response: If no reply is received after multiple retransmissions, or if an ICMP Type 3 unreachable message is returned, the port is marked as filtered.

Operational Advantages:

• Stealth at the Application Layer: Because the three-way handshake is never completed, standard application-level socket APIs (e.g., accept()) never pass the connection to listening server software (such as Apache, Nginx, or OpenSSH). Consequently, the scan generates no application-layer log entries in web or service access logs, though network firewalls and NIDS/IPS appliances will detect the anomalous SYN/RST pattern unless throttled.
• Speed: The scanner does not wait for connection establishment or graceful session closure, enabling high-performance scanning.

2. TCP Connect Scan (-sT): The Unprivileged Berkeley Sockets Scan

When a user executes Nmap without administrative (root) privileges, the operating system kernel prevents the direct creation of raw network packets. In this scenario, Nmap defaults to the TCP Connect Scan (-sT).

Execution Steps and Packet Flow:

1. Nmap invokes the high-level operating system network subsystem using the Berkeley sockets connect() system call.
2. The underlying operating system kernel handles the entire handshake: transmitting the initial SYN, receiving the SYN-ACK, and automatically replying with the final ACK.
3. Once the TCP circuit is fully established, Nmap records the port as open and immediately terminates the connection by issuing a close() system call, which transmits a FIN or RST packet.

Operational Trade-offs:

• No Root Privileges Required: Can be executed by any standard, unprivileged user account on Linux, macOS, or Windows.
• Heavy Logging Footprint: Because the connection is fully completed, the listening application daemon accepts the socket. Web servers, mail servers, and database listeners record the connection event in their application logs, often generating "Connection reset by peer" or empty request error notices.
• Higher Overhead: Managing hundreds of simultaneous full OS socket connections consumes significant CPU cycles and file descriptors on the scanning host, making -sT slower than raw SYN scans.

---

Inverse / Stealth Scans & RFC 793 Compliance

To bypass basic stateless packet inspection firewalls and packet filters monitoring for incoming SYN packets, security researchers developed inverse stealth scans. These scans rely on a specific behavioral mandate defined in RFC 793 (Transmission Control Protocol Specification).

The RFC 793 Rule

RFC 793 (Section 3.9, Page 65) explicitly dictates how a conforming TCP implementation must handle incoming segments that do not contain a SYN, ACK, or RST flag:

• Rule 1 (Closed Port): If the destination port state is CLOSED, an incoming segment not containing a RST causes a RST packet to be sent in response.
• Rule 2 (Open Port): If the destination port state is OPEN, an incoming segment not containing a SYN, ACK, or RST must be silently dropped without generating a response.

                                RFC 793 INVERSE SCANS
[Target Port is CLOSED]
Scanner                                                       Target
   |---------- Probe (NULL, FIN, or Xmas: No SYN/ACK/RST) --->|
   |<--------- TCP RST Packet --------------------------------| (Must Reply with RST)

[Target Port is OPEN]
Scanner                                                       Target
   |---------- Probe (NULL, FIN, or Xmas: No SYN/ACK/RST) --->|
   |           [Packet is Dropped Silently by RFC 793]        | (No Response Received)

The Three Inverse Scan Variants

1. TCP NULL Scan (-sN): Transmits a TCP packet with zero control flags set (all 6 control bits: URG, ACK, PSH, RST, SYN, FIN are 0).
2. TCP FIN Scan (-sF): Transmits a TCP packet with only the FIN (Finish) control flag set.
3. TCP Xmas Scan (-sX): Transmits a TCP packet with the FIN, PSH (Push), and URG (Urgent) flags set simultaneously. The scan is colloquially named the "Christmas tree scan" because the flags are metaphorically "lit up like lights on a Christmas tree."

Port State Interpretation for Inverse Scans

• closed: If a TCP RST packet is received, the port is definitely closed.
• open|filtered: If no response is received after retries, Nmap marks the port as open|filtered. Nmap cannot know whether the packet was dropped because the port is genuinely open (per RFC 793) or dropped because a firewall silently discarded the probe.
• filtered: If an ICMP Type 3 Unreachable error is received (specifically Code 1 [Host Unreachable], Code 2 [Protocol Unreachable], Code 3 [Port Unreachable], Code 9 [Destination Network Administratively Prohibited], Code 10 [Destination Host Administratively Prohibited], or Code 13 [Communication Administratively Prohibited]), the port is marked as filtered.

Critical Operating System Non-Compliance: The Windows Caveat

The fundamental limitation of inverse scans lies in operating system compliance. RFC 793 is an internet standard, but several major operating systems violate RFC 793 implementation guidelines:

The Windows / Cisco Non-Compliance Rule: Microsoft Windows (all desktop and server releases), Cisco IOS, BSDI, and several embedded appliances do not adhere to RFC 793. When an inverse probe (NULL, FIN, or Xmas) arrives at an open port on a Windows host, the Windows TCP/IP stack sends a RST packet regardless of whether the port is open or closed.

Penetration Testing Impact: If an analyst runs an inverse scan (-sN, -sF, -sX) against a Microsoft Windows server hosting active, open services (such as IIS on port 80 or SMB on port 445), Nmap will report every single port as closed. A penetration tester who observes that an entire host returns 100% closed ports on an inverse scan should immediately suspect a Windows-based target or an inline device resetting traffic.

---

Specialized TCP Scans: ACK and Window

1. TCP ACK Scan (-sA)

• Mechanics: Transmits raw TCP packets with only the ACK flag set.
• Target Response Behavior: Unsolicited ACK packets have no corresponding entry in the target's connection table. Therefore, per RFC 793, both open and closed ports must return a RST packet.
• Operational Purpose: The ACK scan never identifies open ports. It is used exclusively as a firewall ruleset mapping tool:
  • unfiltered: If Nmap receives a RST packet, it confirms that the ACK probe successfully reached the host and the return RST reached the scanner. The port is unfiltered (no firewall is blocking traffic).
  • filtered: If no response is received, or if an ICMP Type 3 error is returned, the port is filtered by an intervening stateful firewall or packet filter.

2. TCP Window Scan (-sW)

• Mechanics: Transmits the exact same ACK probe as the ACK scan, but inspects the TCP Window Size field inside the returning RST packet.
• Operating System Quirk: On specific operating systems (including older versions of FreeBSD, OpenBSD, NetBSD, IRIX, and AIX), an implementation anomaly causes open ports to return a RST with a positive, non-zero Window size, while closed ports return a RST with a zero Window size.
• Limitation: On modern operating systems that return zero or arbitrary non-zero Window values uniformly, -sW is unreliable and falsely reports open ports as closed or vice versa.

---

UDP Port Scanning Mechanics (-sU)

The User Datagram Protocol (UDP, RFC 768) is a connectionless, best-effort transport protocol. UDP possesses no three-way handshake, no sequence numbers, and no connection state.

                                 UDP SCAN (-sU)
[Target Port is CLOSED]
Scanner                                                       Target
   |----------------- UDP Datagram (Port 53) ----------------->|
   |<---------- ICMP Port Unreachable (Type 3, Code 3) --------| (Definitively Closed)

[Target Port is OPEN and Service Replies]
Scanner                                                       Target
   |----------------- UDP Datagram (DNS Query) --------------->|
   |<---------------- UDP Response (DNS Answer) ---------------| (Definitively Open)

[Target Port is OPEN (No Payload Sent) OR FILTERED]
Scanner                                                       Target
   |----------------- Empty UDP Datagram --------------------->|
   |                 [No Response / Dropped]                  | (Marked: open|filtered)

Response States in UDP Scanning

1. Closed Port: The target kernel returns an ICMP Type 3, Code 3 (Destination Unreachable: Port Unreachable) error packet.
2. Open Port (Responsive): If Nmap transmits a protocol-specific payload (such as a DNS status query to port 53 or an SNMP community string to port 161) and the application replies with a UDP packet, the port is marked as open.
3. Open|Filtered: If an empty UDP probe is sent to an open port, most services discard the empty datagram without replying. Because an open port dropping a packet and a firewall filtering a packet produce the exact same outcome (silence), Nmap marks the port as open|filtered.
4. Filtered: If an ICMP Type 3 Unreachable error with Codes 1, 2, 9, 10, or 13 is received, the port is marked as filtered.

The UDP Scanning Bottleneck: RFC 1812 ICMP Rate Limiting

Penetration testers frequently observe that UDP scans execute at an agonizingly slow pace compared to TCP scans. The technical root cause is ICMP error rate limiting.

• RFC 1812 Requirement: Section 4.3.2.8 of RFC 1812 requires routers and hosts to limit the rate at which ICMP error messages (such as Destination Unreachable) are transmitted to prevent denial of service and network flooding.
• Linux Implementation: The standard Linux kernel enforces ICMP rate limiting via the kernel parameter net.ipv4.icmp_ratelimit (defaulting to 1000 milliseconds). Consequently, a Linux server will transmit at most one ICMP Destination Unreachable packet per second.
• Mathematical Impact: Scanning all 65,535 UDP ports against a rate-limited Linux host requires a theoretical minimum of 65,535 seconds—over 18.2 hours!

Optimization Strategies for UDP Scanning:

• Restrict UDP scanning strictly to common high-value UDP ports rather than scanning the full range:

  nmap -sU -p 53,67,68,69,123,161,162,500,514,1194 10.10.10.50
  

• Scan top UDP ports using --top-ports 100.
• Calibrate retransmission parameters using --max-retries 1 and --host-timeout 30m.
• Pair UDP scanning with version detection (-sU -sV), which forces Nmap to send protocol-specific payload probes that elicit direct application responses rather than waiting for ICMP error timeouts.

---

Port Scanning Methods Comparison Reference