Mastering Web Application Content Discovery: A Pentester’s Workflow
When conducting a web application penetration test, finding hidden files, directories, and subdomains is crucial. Relying on a single approach is never enough; a thorough assessment requires combining manual analysis, OSINT, and automated enumeration.
Here is a breakdown of the three-pillar workflow every security researcher should follow before moving to the exploitation phase:
1. Manual Reconnaissance (Quick Wins)
Manual checks allow you to spot obvious entry points and configuration oversights quickly. Key areas to inspect include:
- Robots.txt & Sitemap.xml: Uncovers paths the webmaster tried to hide from search engines.
- HTTP Headers & Framework Stack: Identifies the underlying technologies, server types, and potential security misconfigurations.
- Favicon Fingerprinting: Helps determine the exact software or framework running behind the application.
2. OSINT (Open Source Intelligence)
OSINT helps gather publicly available information without directly touching or alerting the target server:
- Google Dorking: Uses advanced search operators to find sensitive files or exposed directories.
- Wappalyzer & Wayback Machine: Analyzes historical snapshots and identifies technologies or outdated endpoints.
- GitHub & S3 Buckets: Searches for leaked API keys, credentials, or misconfigured cloud storage.
3. Automated Discovery (Gobuster Fundamentals)
When manual and OSINT techniques reach their limit, automated tools cover the breadth required to find deeply hidden assets. Using Gobuster, testers typically utilize three core modes:
- Directory Mode (dir): Brute-forces hidden files and folders using wordlists like SecLists, filtering out false positives via status codes (e.g., finding /assets, /private, or log files).
- DNS Mode (dns): Enumerates subdomains to find potential vulnerabilities sitting on secondary endpoints (e.g., shop.example.thm), handling wildcard DNS configurations when necessary.
- Virtual Host Mode (vhost): Sends HTTP requests cycling through a wordlist in the Host header to discover hidden virtual hosts running on the same IP address.
Conclusion
A comprehensive content discovery workflow integrates all three methods seamlessly. The directories, files, and subdomains uncovered during this reconnaissance phase feed directly into the later stages of a penetration test, mapping out the entire attack surface.
How does this look for your blog post? Let me know if you would like to tweak the tone or add any specific details!