# robots.txt # area to be restricted User-agent: * Disallow: /private/ Disallow: /mailers/ Disallow: /land/ Disallow: /Newsletter/ Disallow: /css/ Disallow: /js/ Disallow: /res/ # Disallow / # Disallow /dir/ # Disallow /dir&file # Disallow /xxx.htm User-agent: NetMechanic Disallow: /private/ User-agent: Wells Search II Disallow: /private/ Disallow: /landing_pages/