10.2 Web Protocols and Historical Retrieval Systems

10.2 Web Protocols and Historical Retrieval Systems

The World Wide Web is not synonymous with the Internet; rather, the Web is an application-layer system constructed on top of the underlying Internet networking infrastructure. Before the Web emerged as the dominant medium for digital communication, early network researchers developed a diverse array of text-based retrieval protocols and terminal interfaces. Understanding these foundational protocols, the architecture of modern web transmission standards, and the governance of open web specifications is critical for web development educators and network systems specialists.

---

The Transport Layer Foundation: TCP versus UDP

Application-layer web protocols do not transmit raw data bits directly across physical wires or radio waves. Instead, they rely on the Transport Layer (Layer 4 of the OSI model, or the Host-to-Host layer in the TCP/IP stack) to manage communication channels between software processes running on remote hosts.

TCP: 3-WAY HANDSHAKE (Reliable, Ordered, Connection-Oriented)
Client                                Server
  │                                     │
  ├────────── 1. SYN (Seq=100) ────────>│
  │                                     │
  │<─── 2. SYN-ACK (Seq=300, Ack=101) ──┤
  │                                     │
  ├────────── 3. ACK (Ack=301) ────────>│
  │                                     │
  ▼     [Connection Established]        ▼

UDP: CONNECTIONLESS FIRE-AND-FORGET (Low Latency, Unordered)
Client                                Server
  │                                     │
  ├──────── Datagram 1 (Data) ─────────>│
  ├──────── Datagram 2 (Data) ─────────>│  (No handshake; lost packets
  ├──────── Datagram 3 (Data) ─────────>│   are never retransmitted)
  ▼                                     ▼

Transmission Control Protocol (TCP)

Governed by RFC 9293, TCP (Transmission Control Protocol) is a connection-oriented, highly reliable transport protocol designed to guarantee that data arrives without corruption, duplication, or missing segments.

• The Three-Way Handshake: Before transmitting a single byte of application data, TCP establishes a verified synchronization channel between client and server:
  1. SYN (Synchronize): The client generates an initial sequence number ($x$) and transmits a SYN segment to the server's listening port.
  2. SYN-ACK (Synchronize-Acknowledgment): The server allocates buffer memory, generates its own sequence number ($y$), and transmits an acknowledgment number ($x + 1$) alongside its SYN flag.
  3. ACK (Acknowledge): The client returns an acknowledgment segment containing ($y + 1$). The virtual connection is now established, and application data transfer begins.
• Packet Sequencing and Integrity: TCP slices large application payloads into discrete segments, assigning sequential numbers to each. If network congestion routes packets out of sequence, the receiving host uses these sequence numbers to reconstruct the original data stream in exact chronological order.
• Flow Control and Reliability: TCP implements positive acknowledgment with retransmission (PAR). If the sender does not receive an acknowledgment (ACK) within a specified round-trip timeout window, the missing segment is automatically retransmitted. Through a dynamic sliding window algorithm, TCP throttles transmission rates to match the receiving host's processing capacity, preventing network buffer saturation.
• Primary Web Use Cases: HTTP/1.1, HTTP/2, HTTPS, SFTP, SSH, and SMTP email transfer—protocols where data completeness is non-negotiable.

User Datagram Protocol (UDP)

Governed by RFC 768, UDP (User Datagram Protocol) is a connectionless, lightweight, unreliable transport protocol. In networking terminology, "unreliable" does not imply poor quality; rather, it indicates that the protocol provides no built-in operational guarantees regarding packet delivery, packet ordering, or duplicate packet suppression.

• Zero Handshake Overhead: UDP transmits standalone packets (datagrams) immediately without negotiating a preliminary connection. Each datagram carries minimal header overhead (8 bytes for UDP versus a minimum of 20 bytes for TCP).
• No Retransmission or Flow Control: If network noise or router queue overflows cause datagrams to drop, UDP makes no attempt to recover them. Arriving packets are not re-sequenced.
• Primary Web Use Cases: Real-time interactive multimedia, VoIP teleconferencing, live video streaming, multiplayer gaming network loops, and DNS queries (port 53), where instantaneous data delivery takes precedence over perfect fidelity. (Note: HTTP/3 is built on QUIC, an advanced transport protocol operating over UDP that implements user-space reliability and stream multiplexing).

---

Core Web Application Protocols: HTTP versus HTTPS

At the application layer, the World Wide Web is powered by the Hypertext Transfer Protocol (HTTP), originally documented by Tim Berners-Lee in 1991.

HTTP Architecture and Mechanics

HTTP operates on default port 80 as a stateless request-response protocol. Statelessness means that the web server processes each incoming HTTP request as an entirely independent transaction, preserving no persistent memory of prior interactions with that client. To overcome this limitation and maintain user sessions in web applications (such as student login credentials or interactive shopping carts), developers employ auxiliary state-management mechanisms including HTTP cookies, JSON Web Tokens (JWTs), and server-side session databases.

Core HTTP Request Methods (Verbs)

HTTP defines explicit request methods indicating the desired action to be performed on the target resource:

• GET: Requests a representation of the specified resource. GET requests must be safe (producing no server-side state mutations) and idempotent (repeated identical requests yield the same outcome). Input parameters are encoded directly into the URL query string, making them visible in browser history and server access logs.
• POST: Submits data payload (such as HTML form submissions, user logins, or file uploads) to be processed by the identified server-side script. POST requests are non-idempotent and carry their payload inside the HTTP message body.
• PUT: Uploads a payload designed to replace the entire current representation of the destination resource at that exact URI. PUT is idempotent.
• DELETE: Deletes the specified resource from the server file system or database. DELETE is idempotent.
• PATCH: Applies partial modifications to an existing resource, rather than replacing the complete entity.

Standard HTTP Response Status Codes

Servers communicate the outcome of a request using three-digit numerical status codes organized into five functional categories:

HTTP STATUS CODE TAXONOMY:
1xx: Informational   ──> Protocol negotiation in progress (e.g., 101 Switching Protocols)
2xx: Success         ──> Action successfully received, understood, and accepted
3xx: Redirection     ──> Further action needed by user agent to fulfill request
4xx: Client Error    ──> Request contains invalid syntax or cannot be fulfilled
5xx: Server Error    ──> Server encountered an internal failure fulfilling valid request

• 200 OK: Standard successful HTTP response; the requested resource is enclosed in the response body.
• 201 Created: The request was successful and resulted in the generation of a new resource (standard response for API POST operations).
• 301 Moved Permanently: Signals that a resource has a permanent new URI. Browsers and intermediaries may cache the redirect, and search engines generally consolidate indexing signals toward the destination over time; neither bookmark rewriting nor ranking transfer is guaranteed.
• 302 Found (Temporary Redirect): Signals a temporary alternate URI. Clients follow the redirect, while search engines commonly keep the original URI as canonical, subject to their own crawling and indexing decisions.
• 304 Not Modified: Sent in response to conditional cache validation requests (If-Modified-Since or If-None-Match), informing the browser that its locally cached copy remains valid, saving bandwidth.
• 400 Bad Request: The server cannot process the request due to malformed request syntax, invalid routing framing, or deceptive request routing.
• 401 Unauthorized: Authentication is required and has failed or has not yet been provided.
• 403 Forbidden: The server understands the request and the client's identity, but explicitly refuses authorization (e.g., directory listing disabled, or user lacks role-based permissions).
• 404 Not Found: The server cannot find a resource matching the requested URI.
• 500 Internal Server Error: A generic catch-all error indicating that the server encountered an unexpected condition or unhandled scripting exception that prevented it from fulfilling the request.
• 502 Bad Gateway: A gateway or reverse proxy server received an invalid response from an upstream backend server.
• 503 Service Unavailable: The server is currently incapable of handling the request due to temporary system overload or scheduled maintenance.

HTTPS: Securing the Application Layer

Plaintext HTTP transmits headers, cookies, and data payloads in unencrypted text across intermediate routers, making it highly vulnerable to packet sniffing, credential theft, and Man-in-the-Middle (MitM) content injection. HTTPS (HTTP Secure) resolves this vulnerability by running HTTP over an encrypted Transport Layer Security (TLS) tunnel on default port 443.

HTTPS delivers three foundational cryptographic guarantees:

1. Confidentiality (Encryption): All communication between client browser and web server is encrypted using robust symmetric ciphers (such as AES-256 or ChaCha20) negotiated through an initial asymmetric public-key exchange. Intermediate network snoopers perceive only scrambled ciphertext.
2. Authentication: A valid X.509 certificate and hostname check bind the TLS session to the domain named in the certificate and help detect an impersonating endpoint. Ordinary domain-validated certificates prove domain control, not that the operator is trustworthy or that the site cannot be used for phishing.
3. Data Integrity: Authenticated-encryption tags or message-authentication codes allow the endpoints to detect in-transit alteration. TLS cannot prevent a trusted endpoint itself from serving malicious content.

---

File Transfer and Terminal Administration: Cleartext versus Encrypted Protocols

Beyond serving web documents, server administrators require mechanisms to upload web files and administer server operating systems remotely.

UNENCRYPTED LEGACY PROTOCOLS (Insecure over public networks):
- FTP (Ports 20/21): Usernames, passwords, and file contents sent in plaintext.
- Telnet (Port 23): Terminal keystrokes and administrative passwords sent in plaintext.

ENCRYPTED MODERN STANDARDS (Mandatory for production administration):
- SFTP (Port 22): Subsystem of SSH; full encryption of authentication and file streams.
- FTPS (Ports 21/990): FTP wrapped in TLS; requires complex multi-port firewall rules.
- SSH (Port 22): Cryptographically secured remote terminal access and shell automation.

FTP versus SFTP and FTPS

• FTP (File Transfer Protocol): Dating back to 1971 (RFC 114), FTP operates across two separate ports: port 21 for sending command instructions and port 20 for streaming data payloads. Crucially, FTP contains zero native encryption; usernames, administrative passwords, and source code files travel across networks in unencrypted ASCII text. Anyone operating a packet analyzer (e.g., Wireshark) on the local network path can capture credentials effortlessly.
• FTPS (FTP over SSL/TLS): Adds transport encryption to traditional FTP by encapsulating the control and data channels within TLS. However, because FTPS retains the dual-channel architecture of FTP (using arbitrary ephemeral data ports), configuring corporate and school district firewalls to support FTPS without introducing security vulnerabilities is notoriously difficult.
• SFTP (SSH File Transfer Protocol): Despite sharing "FTP" in its acronym, SFTP is an entirely distinct protocol designed from the ground up as a secure subsystem of SSH (Secure Shell). SFTP runs as an SSH subsystem and commonly uses SSH's default port 22, although an administrator can configure SSH on another port. Commands and file data travel through the encrypted SSH connection. Because it uses a single encrypted SSH connection and supports public-key authentication, SFTP is a common choice for publishing web files to remote servers; HTTPS-based deployment APIs, version-control workflows, and other managed transfer systems are also widely used.

Telnet versus SSH

• Telnet: Developed in 1969 under RFC 854, Telnet provides bidirectional, interactive text-oriented communications over port 23. Telnet transmits terminal traffic, including credentials, in cleartext. It is unsuitable for administration across untrusted or production networks; limited legacy use should be isolated and replaced with an encrypted protocol such as SSH.
• SSH (Secure Shell): Developed in 1995 to replace Telnet, rlogin, and rsh, SSH operates on port 22. It employs asymmetric public-key cryptography to authenticate servers and clients, establishing an encrypted symmetric tunnel that protects all terminal commands, script executions, and administrative tasks against network interception.

---

Historical Information Retrieval Architectures

Prior to the 1990s, the Internet existed as a text-only research network connecting academic, military, and corporate computer labs. Retrieving information required remembering precise server IP addresses, mastering idiosyncratic command-line utilities, and navigating complex directory trees.

PRE-WEB INFORMATION RETRIEVAL MILESTONES:

1969: ARPANET / Telnet (Port 23) ──> Text terminal remote shell access
1971: FTP (Ports 20/21)          ──> Direct remote file transfer
1990: Archie                     ──> First search indexing engine for anonymous FTP archives
1991: Gopher (Port 70)           ──> Menu-driven, hierarchical text retrieval (Univ. of Minn.)
1991: WAIS                       ──> Full-text indexed database search with relevance scoring
1992: Veronica                   ──> Keyword search engine indexing menu titles across Gopherspace
1989-1991: World Wide Web (CERN) ──> Tim Berners-Lee merges Hypertext + TCP + DNS (HTML/HTTP/URL)

The Pre-Web Landscape: Gopher, WAIS, Archie, and Veronica

1. Gopher: Developed in the spring of 1991 by Mark P. McCahill and his team at the University of Minnesota (named after the university's mascot), Gopher is a document retrieval protocol operating on port 70. Gopher presented information as a strict, hierarchical tree of nested menus. Users navigated menus using keyboard arrow keys, drilling down from campus-level directories to individual text files, sound clips, or telnet links. Unlike the Web, Gopher was strictly menu-driven; it could not display inline graphics or embed interactive hypermedia links directly within paragraphs of body text.
2. WAIS (Wide Area Information Servers): Invented in 1991 by Brewster Kahle and Thinking Machines Corporation, WAIS was an early networked text search system based on the Z39.50 information retrieval standard. WAIS indexed the full text of documents stored across distributed databases, allowing users to execute natural language keyword queries. Remarkably, WAIS implemented relevance feedback ranking, enabling researchers to refine search queries by flagging previously retrieved documents as relevant.
3. Archie: Created in 1990 by Alan Emtage, Bill Heelan, and Peter J. Deutsch at McGill University, Archie is recognized as the world's first Internet search engine. Archie did not index web pages (which did not yet exist); instead, it periodically connected to anonymous FTP sites around the world, downloaded directory listings of public files, and compiled them into a searchable local index, allowing users to locate specific software binaries and documents across the global Internet.
4. Veronica (Very Easy Rodent-Oriented Net-wide Index to Computerized Archives): Developed in 1992 at the University of Nevada, Reno, Veronica served as the search engine for Gopherspace. It continually scanned Gopher servers worldwide, harvesting the titles of menu items and building a centralized database that allowed users to perform Boolean keyword searches across thousands of interconnected Gopher menus.

The Invention of the World Wide Web

Between 1989 and 1991, British computer scientist Tim Berners-Lee, working at the CERN physics laboratory in Geneva, Switzerland, conceptualized a revolutionary solution to the information fragmentation plaguing international research collaborations. In his historic 1989 proposal, "Information Management: A Proposal," Berners-Lee synthesized three previously distinct technological concepts into a unified architecture:

1. Hypertext: Non-linear document linking, formulated through HTML (Hypertext Markup Language), allowing any word, phrase, or graphic to function as an interactive portal linking directly to any other document.
2. Universal Addressing: The URI/URL (Uniform Resource Identifier / Locator), providing an unambiguous, globally unique address for every information resource on the planet.
3. Internet Transport: The HTTP (Hypertext Transfer Protocol), a lightweight, stateless request-response protocol designed to fetch hypertext files across TCP/IP networks via DNS name resolution.

Berners-Lee authored the first web server (CERN httpd) and the first graphical web browser-editor (WorldWideWeb, later renamed Nexus) on a NeXTcube computer, releasing the Web to the global public in August 1991. By uniting hypertext linking with internetworking, the Web bypassed the rigid hierarchical boundaries of Gopher and FTP, sparking the modern digital age.

---

Web Standards Organizations and Governance

The ongoing interoperability of the World Wide Web depends upon open, vendor-neutral technical standards that prevent proprietary lock-in.

• W3C (World Wide Web Consortium): Founded in 1994 by Tim Berners-Lee at MIT, the W3C serves as the international standards organization for the Web. Governed by member organizations and full-time engineering staff, the W3C produces formal Recommendations that define HTML, CSS, SVG, XML, and accessibility standards. Crucially, the W3C's Web Accessibility Initiative (WAI) authors the Web Content Accessibility Guidelines (WCAG), establishing legal and technical standards for accessible web content.
• WHATWG (Web Hypertext Application Technology Working Group): Formed in 2004 by engineers from Apple, Mozilla, and Opera, the WHATWG arose in response to the W3C's decision to abandon HTML in favor of XML-strict XHTML. The WHATWG advocated for an evolving, backward-compatible standard known as HTML5. In 2019, the W3C and WHATWG signed a formal partnership agreement, recognizing the WHATWG's HTML Living Standard as the sole authoritative version of HTML.
• Ecma International: An international industry association dedicated to the standardization of information and communication systems. Ecma oversees TC39 (Technical Committee 39), the engineering body that authors and maintains ECMA-262—the official standard governing ECMAScript (the core specification implemented by all commercial JavaScript runtimes).

---

Historical and Modern Internet Protocols Comparison