Web Reconnaissance

HTTP Headers Reference

Request Headers (Useful for Recon)

Header	Description
Host	Target hostname - useful for vhost discovery
User-Agent	Client identifier - spoofable to bypass restrictions
Cookie	Session tokens, auth data
Authorization	Basic/Bearer tokens
Referer	Previous page - spoofable for access bypasses
X-Forwarded-For	Client IP (via proxy) - spoofable for IP bypasses

Response Headers (Look For)

Header	Description
Server	Web server software & version
X-Powered-By	Backend technology (PHP, ASP.NET, Express)
X-Redirect-By	CMS identification (WordPress, Drupal)
Set-Cookie	Session handling, flags (Secure, HttpOnly)
Content-Type	Response type and encoding
WWW-Authenticate	Auth mechanism required

Security Headers (Missing = Potential Issue)

Header	Description
Content-Security-Policy	XSS prevention - script sources
X-Frame-Options	Clickjacking protection
Strict-Transport-Security	HTTPS enforcement (HSTS)
X-Content-Type-Options	Prevent MIME sniffing
Referrer-Policy	Control referrer information
Permissions-Policy	Feature restrictions

---

Web Fingerprinting

Identify Server & Tech Stack (curl)

curl -I http://TARGET
curl -I https://TARGET

# Follow redirects
curl -IL http://TARGET

Look for:

• Server: header (e.g., Apache/2.4.41, nginx/1.18.0)

• X-Powered-By: header (e.g., PHP/7.4, Express)

• X-Redirect-By: header (e.g., WordPress)

---

WAF Detection (wafw00f)

# Install
pip3 install git+https://github.com/EnableSecurity/wafw00f

# Detect WAF
wafw00f TARGET

---

Nikto Fingerprinting

# Install
sudo apt update && sudo apt install -y perl
git clone https://github.com/sullo/nikto
cd nikto/program && chmod +x ./nikto.pl

# Fingerprint only (no vuln scan)
nikto -h TARGET -Tuning b

---

robots.txt Analysis

curl http://TARGET/robots.txt

Key Directives

Directive	Description
Disallow:	Paths the bot shouldn't crawl (interesting for recon!)
Allow:	Explicitly permitted paths
Sitemap:	URL to sitemap.xml
Crawl-delay:	Seconds between requests

Recon Value: Disallowed paths often reveal admin panels, backup directories, or sensitive endpoints.

---

.well-known URIs

# Security contacts
curl http://TARGET/.well-known/security.txt

# OpenID Connect config (OAuth endpoints)
curl http://TARGET/.well-known/openid-configuration

# Password change URL
curl http://TARGET/.well-known/change-password

Useful .well-known URIs

URI	Description
security.txt	Security contact info
openid-configuration	OAuth2/OIDC endpoints, supported scopes
assetlinks.json	Android app asset links
mta-sts.txt	Email MTA-STS policy

Full list: https://www.iana.org/assignments/well-known-uris/well-known-uris.xhtml

---

Google Dorking

Operators

Operator	Description	Example
site:	Limit to domain	site:example.com
inurl:	Term in URL	inurl:admin
intitle:	Term in page title	intitle:"index of"
filetype:	Specific file type	filetype:pdf
ext:	File extension	ext:conf
intext:	Term in page body	intext:password
cache:	Cached version	cache:example.com
link:	Pages linking to URL	link:example.com

Common Dorks

# Login pages
site:TARGET inurl:login
site:TARGET (inurl:login OR inurl:admin)

# Exposed files
site:TARGET filetype:pdf
site:TARGET (filetype:xls OR filetype:docx)
site:TARGET filetype:sql

# Config files
site:TARGET inurl:config.php
site:TARGET (ext:conf OR ext:cnf OR ext:ini)
site:TARGET ext:env

# Backups
site:TARGET inurl:backup
site:TARGET (ext:bak OR ext:old OR ext:backup)

# Directory listings
site:TARGET intitle:"index of"

# Passwords/secrets
site:TARGET intext:password filetype:txt
site:TARGET intext:api_key

Google Hacking Database

https://www.exploit-db.com/google-hacking-database

---

Wayback Machine

Web Interface

https://web.archive.org/

Enter URL to view historical snapshots.

Command Line

# Fetch all archived URLs for a domain
curl "https://web.archive.org/cdx/search/cdx?url=*.TARGET&output=txt&fl=original&collapse=urlkey" | sort -u

# View specific snapshot
https://web.archive.org/web/TIMESTAMP/http://TARGET/

Recon Value

• Find old pages, endpoints, files no longer linked

• Discover removed sensitive content

• Track tech stack changes over time

• Passive (no direct target interaction)

---

Web Crawling

Burp Suite Spider

1. Set scope to target domain

2. Right-click target in Site Map → Spider this host

3. Review discovered endpoints in Site Map

OWASP ZAP Spider

1. Set target URL

2. Right-click → Attack → Spider

3. Review Sites tree for discovered content

ReconSpider (Custom Tool)

wget -O ReconSpider.zip https://academy.hackthebox.com/storage/modules/144/ReconSpider.v1.2.zip
unzip ReconSpider.zip
python3 ReconSpider.py http://TARGET

Output in results.json:

• Emails, links, external files

• JavaScript files

• Form fields

• Images, videos, audio

• HTML comments

Scrapy (Python)

pip3 install scrapy

Custom spider for large-scale crawling.

---

Automation Frameworks

Tool	Description
FinalRecon	Python-based, modular (SSL, WHOIS, headers, crawl)
Recon-ng	Framework with modules for DNS, subdomains, ports, etc.
theHarvester	Email, subdomain, host gathering from multiple sources
Amass	Comprehensive subdomain enumeration
Photon	Fast crawler extracting URLs, emails, files, endpoints

FinalRecon

git clone https://github.com/thewhiteh4t/FinalRecon.git
cd FinalRecon
pip3 install -r requirements.txt

python3 finalrecon.py --full http://TARGET

Recon-ng

# Start framework
recon-ng

# Create workspace
workspaces create TARGET

# Add target domain
db insert domains TARGET.com

# Use module
modules load recon/domains-hosts/hackertarget
run

# View results
show hosts