A penetration tester conducts OSINT for a client and discovers the robots.

In OSINT, the tester’s objective is to gather information using publicly available, non-intrusive sources without altering the target environment. A robots.txt file is a crawler directive that can discourage or block specific search engine bots from indexing certain paths. If one “major” engine is explicitly blocked, the most practical OSINT adjustment is to use other search engines and data providers that may not be restricted in the same way or may already have historical indexing and cached results. This aligns with PenTest+ reconnaissance techniques that emphasize using multiple sources and pivoting between providers to maximize coverage and reduce blind spots.

Modifying the WAF or changing robots.txt would be active alteration of the client’s systems and is not an OSINT method; it also typically falls outside the intent of passive recon and may violate rules of engagement. A CSRF attack is an exploitation technique unrelated to discovering publicly indexed information. Therefore, leveraging a competing provider is the best way to continue OSINT collection when one crawler is blocked.