10.3 Web Reconnaissance, Content Discovery & Threat Modelling

10.3 Web Reconnaissance, Content Discovery & Threat Modelling

A comprehensive web application penetration test begins with structured reconnaissance. Before attempting targeted exploitation, an analyst must map the target's attack surface, enumerate hidden content, identify the underlying technology stack, and construct an architectural threat model. Flaws in web applications frequently originate not only from coding errors in published pages, but from unlinked administrative interfaces, forgotten backup files, exposed configuration files, and unvalidated trust boundaries.

---

Web Application Reconnaissance & Spidering

Reconnaissance against web applications combines passive discovery (analyzing publicly accessible and linked resources) with active forced browsing (probing for unlinked assets using dictionary brute-forcing).

+-----------------------------------------------------------------------------+
|                   WEB APPLICATION DISCOVERY METHODOLOGY                     |
+-----------------------------------------------------------------------------+
| 1. PASSIVE SPIDERING / CRAWLING                                             |
|    - Traverse visible HTML DOM links (<a href>, <form action>, <iframe>)    |
|    - Extract endpoints from client-side JavaScript files (.js)              |
|    - Retrieve JavaScript Source Maps (.js.map) to recover original source   |
|    - Inspect robots.txt, sitemap.xml, /.well-known/, HTML comments          |
+-----------------------------------------------------------------------------+
                                      |
                                      v
+-----------------------------------------------------------------------------+
| 2. ACTIVE FORCED BROWSING & FUZZING                                         |
|    - Dictionary brute-forcing of directories and filenames                  |
|    - Tools: ffuf, gobuster, dirsearch, dirb, wfuzz                          |
|    - Targeted extension fuzzing (.php, .aspx, .json, .bak, .old, .env)      |
|    - Analyze response status codes (200, 301, 302, 401, 403 vs 404)         |
+-----------------------------------------------------------------------------+
                                      |
                                      v
+-----------------------------------------------------------------------------+
| 3. TECHNOLOGY FINGERPRINTING & THREAT MODELLING                             |
|    - HTTP Response Headers (Server, X-Powered-By, X-AspNet-Version)         |
|    - Cookie naming conventions (PHPSESSID, JSESSIONID, ASP.NET_SessionId)   |
|    - CMS indicators (WordPress /wp-admin/, Drupal core, Joomla paths)       |
|    - Map components to STRIDE categories across trust boundaries            |
+-----------------------------------------------------------------------------+

1. Passive Spidering and JavaScript Analysis

• Automated Crawling: Web crawlers (such as the Burp Suite Spider/Crawler or OWASP ZAP Spider) parse HTML responses, recursively populating the target site tree by extracting URLs from <a href="...">, <form action="...">, <script src="...">, <link href="...">, and <img src="..."> tags.
• Analyzing Single Page Applications (SPAs): Modern front-end applications built with frameworks like React, Angular, or Vue bundle their routing logic into client-side JavaScript packages. Analysts inspect these .js files using regular expressions to extract unlinked API routes, development endpoints, and internal microservice addresses.
• JavaScript Source Maps (.map files): During production builds, developers often minify and bundle code. If they inadvertently deploy .map files (e.g., main.bundle.js.map) to the web server, an analyst can use browser developer tools or utilities like sourcemapper to completely de-minify and reconstruct the developer's original raw source code—including developer comments, internal variable names, and unreleased feature endpoints.

2. Informational Files and Metadata

Web administrators deploy standard files intended for crawlers, search engines, or security researchers that inadvertently leak sensitive operational paths:

---

Forced Browsing & Content Discovery

Because crawlers only find resources linked within HTML and JavaScript, security analysts use forced browsing (content discovery) to identify unlinked files, administrative interfaces, and backup copies.

# Directory and endpoint fuzzing using ffuf
ffuf -u https://target.lan/FUZZ -w /usr/share/seclists/Discovery/Web-Content/raft-medium-directories.txt \
     -mc 200,301,302,401,403 -c -v

# Targeted extension fuzzing with Gobuster
gobuster dir -u https://target.lan -w /usr/share/seclists/Discovery/Web-Content/common.txt \
             -x php,aspx,jsp,html,txt,json,env,bak,old,zip -t 40

Interpreting Response Codes During Discovery

• 200 OK: File or directory exists and is publicly accessible.
• 301 / 302 Redirect: Resource exists; check the Location response header. If it redirects to /login or appends a trailing slash (/admin -> /admin/), the directory is verified.
• 401 Unauthorized: Resource exists and is protected by HTTP Basic/Digest/Bearer authentication.
• 403 Forbidden: Resource exists, but access is restricted by server configuration or IP whitelisting. This confirms the presence of an administrative portal or restricted endpoint.
• 404 Not Found: Resource does not exist (ensure the server does not return custom "soft 404" pages with HTTP 200 codes).

Targeted Extension Fuzzing & Backup File Enumeration

When developers edit or back up files directly on production servers, text editors and scripts often create residual artifacts accessible to any user:

+-----------------------------------------------------------------------------+
|                      HIGH-VALUE BACKUP & CONFIG ARTIFACTS                   |
+-----------------------------------------------------------------------------+
| Category          | File Extensions / Filenames                             |
+-------------------+---------------------------------------------------------+
| Web Frameworks    | .php, .asp, .aspx, .jsp, .do, .action, .py, .js         |
| Backup Files      | .bak, .old, .orig, .temp, .backup, .save, ~             |
| Editor Swaps      | .swp, .swo (Vim swap files), .#filename (Emacs)         |
| Configuration     | .env, web.config, httpd.conf, settings.py, database.yml |
| Archives & Dumps  | .zip, .tar.gz, .tar.bz2, .sql, .dump, .7z               |
| Version Control   | /.git/, /.git/HEAD, /.svn/, /.svn/entries, /.hg/        |
+-----------------------------------------------------------------------------+

Critical Vulnerability Pattern: If an Apache web server is configured to execute .php files through its PHP interpreter, requesting index.php executes the code on the server. However, if a developer leaves a backup copy named index.php.bak, index.php~, or index.php.old, Apache does not match the .php extension handler and instead serves the raw file as plaintext, exposing all underlying source code, database passwords, and API keys.

---

Identifying Web Technologies & Frameworks

Accurate fingerprinting allows testers to focus on known Common Vulnerabilities and Exposures (CVEs) and architecture-specific weaknesses.

+-----------------------------------------------------------------------------+
|                      TECHNOLOGY FINGERPRINTING MATRIX                       |
+-----------------------------------------------------------------------------+
| Indicator Category | Disclosed Value / Name        | Implied Technology     |
+--------------------+-------------------------------+------------------------+
| Server Header      | Apache/2.4.52 (Ubuntu)        | Apache on Linux        |
|                    | nginx/1.18.0                  | Nginx Web Server       |
|                    | Microsoft-IIS/10.0            | Windows Server IIS     |
+--------------------+-------------------------------+------------------------+
| Application Header | X-Powered-By: Express         | Node.js Express        |
|                    | X-Powered-By: PHP/7.4.30      | PHP Runtime            |
|                    | X-AspNet-Version: 4.0.30319   | ASP.NET Framework      |
+--------------------+-------------------------------+------------------------+
| Cookie Identifier  | PHPSESSID                     | PHP Application        |
|                    | JSESSIONID                    | Java Servlet Container |
|                    | ASP.NET_SessionId             | Microsoft ASP.NET      |
|                    | connect.sid                   | Node.js Express        |
|                    | csrftoken / sessionid         | Python Django          |
+--------------------+-------------------------------+------------------------+
| Distinct CMS Paths | /wp-admin/, /wp-includes/     | WordPress CMS          |
|                    | /administrator/               | Joomla CMS             |
|                    | /core/, CHANGELOG.txt         | Drupal CMS             |
+-----------------------------------------------------------------------------+

Automated Fingerprinting Tools

• whatweb: Command-line reconnaissance utility that identifies content management systems, blogging platforms, JavaScript libraries, web servers, and embedded devices.

  whatweb https://target.lan -v --color=never
  

• Wappalyzer: Browser extension and CLI package that inspects HTML markup, meta tags, script URLs, JavaScript global variables (window.React, window.jQuery), and HTTP headers to generate a detailed technology profile.
• CMS-Specific Scanners: Utilities like wpscan (WordPress) audit plugins, themes, and users via REST API endpoints (/wp-json/wp/v2/users).

---

Threat Modelling for Web Applications: The STRIDE Model

Threat modelling provides a systematic methodology for evaluating application attack surfaces, mapping data flows, and identifying security weaknesses before testing or development.

+-----------------------------------------------------------------------------+
|                        STRIDE THREAT MODELLING MODEL                        |
+-----------------------------------------------------------------------------+
| Threat Category           | Violated Security Property | Web Attack Example |
+---------------------------+----------------------------+--------------------+
| S - Spoofing              | Authenticity               | Session Hijacking, |
|                           |                            | Credential Replay  |
| T - Tampering             | Integrity                  | Parameter Tampering|
|                           |                            | SQL Injection      |
| R - Repudiation           | Non-repudiation            | Action Denied;     |
|                           |                            | Insufficient Logs  |
| I - Information Disclosure| Confidentiality            | Verbose Error Dumps|
|                           |                            | Directory Listing  |
| D - Denial of Service     | Availability               | ReDoS Regex Bomb,  |
|                           |                            | Resource Exhaustion|
| E - Elevation of Privilege| Authorization              | IDOR / BOLA,       |
|                           |                            | Admin Role Bypass  |
+-----------------------------------------------------------------------------+

Data Flow Diagrams (DFDs) and Trust Boundaries

Threat modelling relies on Data Flow Diagrams (DFDs) to map how data moves through an application:

• External Entities: Users, third-party payment gateways, external APIs.
• Processes: Code modules that compute, transform, or route data (e.g., authentication controller, payment handler).
• Data Stores: Locations where data resides (e.g., SQL databases, log files, session caches).
• Data Flows: Directed arrows showing the movement of data between entities, processes, and stores.
• Trust Boundaries: The most critical element in threat modelling. A trust boundary represents the perimeter where data transitions from one trust level to another (e.g., from an untrusted web browser to a DMZ reverse proxy, or from an application server to an internal database). Any data crossing a trust boundary must be treated as untrusted and subjected to strict authentication, authorization, and input validation.