Free since 2005 · No login required
AT

Academic Tutorials

Learn at your own pace

site-mobile-top-banner · 320x50

How do I use a Robots file or do you have an example of one?

A basic, spider can go anywhere file is: User-agent: * Disallow: You can disallow folders or files by putting this into the file: User-agent: * Disallow: /foldername/