3.1 OSI, TCP/IP, Devices, and the Three-Way Handshake

A Security Operations Center (SOC) analyst uses layer models to place evidence, not to decorate a slide. When a packet capture shows a SYN, a SYN-ACK, and a RST from the scanning host, you are looking at transport-layer state. When a switch Content Addressable Memory (CAM) table overflows and frames flood every port, you are looking at data-link behavior. When a router decrements Time To Live (TTL) and forwards to a new next hop, you are looking at the internet layer. This independent study section covers Network Essentials topics named in TryHackMe Security Analyst Level 1 (SAL1) training content: the Open Systems Interconnection (OSI) model, the TCP/IP model, common devices, key transport behavior, and the Transmission Control Protocol (TCP) three-way handshake. OpenExamPrep publishes this as independent prep for learners studying those SAL1 network topics. It does not claim TryHackMe approval, partnership, or official review, and TryHackMe does not publish a SAL1 percentage weight for this topic.

The OSI seven-layer model

The OSI model (ISO/IEC 7498) splits communication into seven layers. Each layer offers a service to the layer above and uses the layer below. Real stacks do not emit seven equal headers. Ethernet (or Wi-Fi) carries an Internet Protocol (IP) packet; that packet carries a TCP segment or a User Datagram Protocol (UDP) datagram; the payload is Hypertext Transfer Protocol (HTTP), Domain Name System (DNS), or Simple Mail Transfer Protocol (SMTP). Session and presentation ideas still help you talk about dialogues and encoding, but Wireshark will not show a distinct “session header” on a normal web fetch.

Layer 7 Application. The program conversation: HTTP, DNS, SMTP, Secure Shell (SSH). This is where URLs, methods, and mail commands live.

Layer 6 Presentation. Syntax: character encodings, compression, and—in textbook drawings—encryption. On today’s internet, Transport Layer Security (TLS) is implemented as an application protocol on TCP (or inside QUIC). Treat “TLS is layer 6” as a teaching shorthand, not a capture fact.

Layer 5 Session. Managing a conversation (setup, checkpoints, teardown of a dialog). Remote Procedure Call (RPC) sessions are the usual example. Web browsers do not add a distinct session header.

Layer 4 Transport. End-to-end delivery: ports, reliability (TCP), or datagrams (UDP). Sequence numbers, acknowledgments, and flags live here.

Layer 3 Network. IP addressing and routing. IPv4, IPv6, Internet Control Message Protocol (ICMP). Routers make forwarding decisions here.

Layer 2 Data Link. Frames, Media Access Control (MAC) addresses, Virtual LAN (VLAN) tags, Address Resolution Protocol (ARP) on Ethernet. Switches decide outbound ports here.

Layer 1 Physical. Bits on copper, fiber, or radio. Hubs and media converters live here. A damaged cable is a layer-1 incident even if the ticket says “the website is down.”

The TCP/IP four-layer model and the mapping

The TCP/IP model (Internet model) matches how hosts and RFCs actually group functions. RFC 1122 describes host requirements using link, internet, transport, and application layers.

Memorize that mapping. Interview and lab questions often ask which OSI layers collapse into TCP/IP application (5–7) and which device is “layer 3” (router) versus “layer 2” (switch).

Trap: a layer-3 switch can route IP between VLANs. A classic access switch only forwards frames by MAC. If a ticket says “the switch dropped the packet,” ask whether the device was bridging or routing before you blame HTTP.

Devices and the layer they act on

A hub repeats layer-1 bits to all ports and is rare on production LANs; collision domains become huge. A Web Application Firewall (WAF) is an application-layer control; later SAL1 topics cover WAF under network and web protection.

Worked placement: a user cannot reach https://intranet.example.com. Ping to the default gateway works, so layers 1–3 to the router are up. A TCP SYN to port 443 gets no SYN-ACK. The firewall log shows deny tcp 10.20.30.40:51990 -> 10.0.5.20:443. That is a layer-4 policy decision, not a broken HTTP application. If SYN-ACK returns and the browser then shows a certificate name mismatch, you have left transport and entered TLS sitting on TCP.

TCP versus UDP

TCP builds a connection: handshake, ordered byte stream, retransmission, congestion control. UDP sends datagrams with no connection state in the protocol. DNS resolvers usually query over UDP/53 and fall back to TCP/53 when the answer is truncated, or for zone transfers. HTTP/1.1 and HTTP/2 ride TCP. HTTP/3 uses QUIC over UDP, so a 2026 capture of UDP/443 can be encrypted web traffic, not a mystery service.

TCP is not “more secure” than UDP. Security comes from TLS, authentication, and filtering. UDP is not automatically “faster” in a way that closes a ticket; it is simpler, so applications that can tolerate loss—or that implement their own reliability—choose it.

Ports live at the transport layer. A host can have thousands of sockets. Well-known server ports (22, 80, 443) identify the service the server offered, not the client’s ephemeral source port (often in 49152–65535). Do not confuse a switch’s physical port with a TCP port number.

TCP flags you must read in a capture

Watch these flags in tcpdump, Wireshark, or a Zeek conn log. The mnemonic is not enough; you need the packet order.

ECE and CWR exist for Explicit Congestion Notification. You will see them less often than SYN, ACK, FIN, and RST in entry-level labs.

Worked handshake (numbers you can replay on paper)

Client 10.20.30.40:51234 connects to server 203.0.113.10:443. Ignore timestamps; watch sequence (seq) and acknowledgment (ack) numbers.

1. Client to server: flags=SYN, seq=1000. No payload. The client is saying it wants a connection and that its sequence starts at 1000.
2. Server to client: flags=SYN,ACK, seq=8000, ack=1001. The server chooses its own sequence (8000) and acknowledges 1000 by adding one.
3. Client to server: flags=ACK, seq=1001, ack=8001. Both sides now have the other’s starting sequence. The connection is ESTABLISHED.

That is the three-way handshake. Only after packet 3 should you expect a full application exchange. Some stacks put TLS ClientHello bytes on the third packet (data-on-ACK). The control flags are still SYN, then SYN-ACK, then ACK.

If packet 2 never arrives, do not open an HTTP ticket. Look at routing, NAT, an asymmetric firewall path, or a silent drop. HTTP status codes do not exist until the application has a socket.

Worked teardown

Graceful close is typically four packets, not three:

1. Client to server: FIN,ACK — the client has no more data.
2. Server to client: ACK — the server received the FIN.
3. Server to client: FIN,ACK — the server is also done (sometimes after a TLS close-notify).
4. Client to server: ACK.

Either side may FIN first. A simultaneous close (both send FIN) is legal. A RST skips the drain: the socket is gone. Firewalls that reject a port often send RST, or they drop silently. An application crash can RST an established session. Do not treat every RST as an attack. Compare TTL and IP identification if you suspect a middlebox forged the RST.

What a SYN scan looks like to an analyst

A SYN scan (half-open scan, the usual -sS style in Nmap) sends SYN probes and refuses to complete the handshake:

• Open port: target replies SYN-ACK; the scanner immediately sends RST. The service process may never accept() a full connection, so the application log can stay quiet while the capture is loud.
• Closed port: target replies RST (or RST-ACK). You never see SYN-ACK.
• Filtered port: no reply, or ICMP unreachable / administratively prohibited. The scanner cannot tell open from drop.

Worked burst from 198.51.100.77 against 10.0.0.8 in about one second:

• Port 22: SYN, SYN-ACK, then RST from 198.51.100.77. SSH is open; no SSH banner exchange.
• Port 23: SYN, then RST from 10.0.0.8. Telnet is closed.
• Port 445: SYN, no response. Server Message Block (SMB) is filtered by a host firewall or network access list.

Analyst tells: many destination ports, one source, SYNs with little or no payload, RSTs from the scanner after SYN-ACK, and no Application Data. A full connect scan completes the handshake and then FINs or RSTs; that is noisier in application logs. A SYN flood is different: huge SYN volume, often spoofed sources, aimed at exhausting half-open state—not mapping a port list.

When a later SOC-simulator alert says “port scan,” reconstruct this sequence. Do not escalate a single SYN to an unused port from a vulnerability scanner’s approved range without checking the exception list. Do not call the SAL1 exam proctored or invent a domain weight; use the packet facts.