How do I use a Robots file or do you have an example of one? A basic, spider can go anywhere file is: User-agent: * Disallow: You can disallow folders or files by putting this into the file: User-agent: * Disallow: /foldername/