############################### # # robots.txt file for this website # # You should add a robots.txt file to the root directory of all your websites to help control the # indexing of your site by robots that ignore the convention. # In this file you specifically list any pages that you DO NOT want crawled and indexed # # addresses all robots by using wild card * # User-agent: * # list folders robots are not allowed to index # # Disallow: /tutorials/ includes ALL FILES AND SUBFOLDERS located within the folder tutorials directory # will not be indexed by the robots if you use this statement # Disallow: /test/ Disallow: /Pictures/ Disallow: /ALLPictures/ Disallow: /NPR/ Disallow: /Commercial/ Disallow: /control/ Disallow: /Helios/ Disallow: /nydgtest/ Disallow: /Pic/ Disallow: /webmd/ Disallow: /nydgtest/ # # # list specific files robots are not allowed to index # # Disallow: /tutorials/meta_tags.html # Disallow: /tutorials/custom_error_page.html # # End of robots.txt file # ###############################