5.3 DNS, Certificate Transparency, and Search Engine Enumeration

Once social and leak OSINT is in the notes, example.com still has to become a list of names, addresses, and URLs. Objective 2.1 groups that expansion under DNS lookups, reverse DNS lookups, certificate transparency logs, search engine analysis/enumeration, cached pages, and HTML scraping. Together they are how testers find the VPN the marketing site never mentioned and the forgotten /old-admin path still sitting in a 2019 archive.

DNS lookups

A DNS lookup asks a resolver for records. For exam purposes, know what each common type is for:

Querying those records at a public recursive resolver does not put a SYN on the web server. It is the usual passive DNS picture. Two traps sit next to that comfort.

Reverse DNS lookups

Reverse DNS uses PTR records. Given 10.4.12.8 in an in-scope CIDR (or a public IP you already know belongs to the client), a PTR of vpn.dev.example.com corroborates the CT finding from the worked scenario. Reverse lookups also cluster hosts: a block of PTRs named cam01-bldg4 versus pos-store-12 tells you camera versus payment — and payment may be excluded. PTR is not guaranteed; many cloud IPs have generic provider names. Absence of PTR is not absence of a host.

Zone transfer (AXFR) as enumeration

A zone transfer asks an authoritative name server to dump the zone. If it succeeds, you receive the full name list the organization intended only for secondaries. That is a classic misconfiguration finding and a gold-mine enumeration technique. It is not the same as an A-record lookup at Google Public DNS. AXFR is a protocol conversation with the client's NS (or a forgotten secondary still allowing transfers to any querier). Classify it as active DNS enumeration. If RoE has not opened yet, do not AXFR; stick to recursive lookups and third-party DNS history (SecurityTrails-style, DNSdumpster, Amass passive sources — tools chapter).

Failed AXFR is normal and not a finding by itself. Open AXFR to the world is the finding. Incremental transfers (IXFR) are the same idea with a smaller payload.

Certificate transparency logs

Certificate transparency (CT) is a public, append-only log of issued TLS certificates. Browsers and CAs participate so a sneaky certificate cannot hide. For a tester, CT is a hostname enumerator. Search a domain (the crt.sh pattern on the exam) and you see every name that appeared as a common name or subject alternative name (SAN) on a logged cert: www, api, dev, vpn.dev.example.com, *.internal.example.com if someone was careless, plus expired and revoked history.

Worked scenario, again: CT lists vpn.dev.example.com and an expired certificate. Passive facts you may write down now:

• A VPN (or at least a VPN-named host) exists or existed.
• Someone let a cert expire — cryptographic hygiene, possibly still serving a leftover name.
• SANs may list more environments (staging, okta, a partner domain).

Facts you may not assume:

• The host is in-scope. Scope is the SoW, not the SAN list.
• The expired cert is still presented. Confirming live TLS is a banner/handshake against the host (active).
• A SAN for payments.partner.net is client-owned. It is probably a processor; exclusions apply.

CT does not decrypt HTTPS. It does not replace Nmap. It does not give you the private key. If a private key leaked in GitHub (previous section), that is the cryptographic flaw; CT only told you the name the key would have matched.

Search engine analysis and enumeration

Search engine analysis/enumeration is deliberate querying — including dorking — to find indexed pages the company did not mean to keep public: site:example.com filetype:pdf, login panels, directory listings, intitle:"index of", exposed /server-status, backup zips, and third-party hosts that mention the brand. Shodan and Censys sit in this family as search-engine-like internet-scan databases: you query their index; you do not scan. From your perspective that is passive. Treat their banners as historical until you confirm.

Stay professional: dorking public indexes is recon. Using a search engine to locate exploit PoCs is research. Running those exploits is Domain 4 and out of this chapter.

Cached pages

Cached pages and archives (search-engine cache, the Wayback Machine) recover content after the live site is "fixed." Forgotten endpoints, old JavaScript with API keys, product version numbers in a 2022 blog, a staging URL that still resolves, and a retired /phpmyadmin path are common. Reading cache or Wayback is passive. Hitting the live URL you just discovered is active and needs to be in-scope.

If the cache shows a key, handle it like a repo secret: redact, escalate rotation, do not replay it for fun.

HTML scraping

HTML scraping is collecting the live (or authorized-lab) markup and linked assets. CompTIA lists it separately from search engines because you are now talking to the origin or its CDN. In-scope scrape targets:

• HTML comments (<!-- todo: remove debug /api/v1/internal -->).
• Hidden fields and extra form parameters the UI does not show.
• JavaScript bundles that list endpoints, feature flags, and sometimes tokens.
• robots.txt and sitemap.xml (also reachable passively if a search engine cached them; fetching live is a request to the site).
• Framework generator tags, generator meta, and versioned static paths.

Scraping is active when aimed at the live application. It is also usually low-and-slow HTTP GET, not a SYN sweep, so RoE that forbids "scanning" may still allow crawling — unless it forbids that too. Read the RoE. Do not confuse a polite scrape with permission to fuzz every parameter; that is later web-attack work.

Information disclosure from this cluster is often the forgotten thing: a hostname only in CT, a path only in Wayback, a comment only in HTML. The tester's job is to merge those lists, drop exclusions, and hand a clean target list to network recon.