WWW wanderers or spiders are programs that traverse many pages in the World Wide Web...

Place robots.txt file in the root of your website with listing of directories that you don't want to be crawled

Place authentication on root directories that will prevent crawling from these spiders

Enable SSL on the restricted directories which will block these spiders from crawling

Place "HTTP:NO CRAWL" on the html pages that you don't want the crawlers to index