robots google

  [DELETED] 18:12 07 Apr 2006

Hi, I have uploaded an xml sitemap to Google and they say my robots file is 404 error, I do not have a robots file as I want no exclusions, how do I write a file that syas this? Also how long do you think it takes to get indexed in Google?Thanks

  [DELETED] 21:06 07 Apr 2006

Use Notepad

type in

User-agent: *


save the file as


then upload this file to the website.

  beynac 22:45 07 Apr 2006

Isn't this the wrong way round? Your text file stops robots, I thought that FOEYDO was asking for a way to encourage them.

  [DELETED] 23:35 07 Apr 2006

User-agent: *

The asterisk (*) in the User-agent field is shorthand for "all robots".


Because nothing is disallowed, everything is allowed.

It allows all robots to crawl all the files and directories.

  beynac 07:17 08 Apr 2006

I think that we will have to agree to differ. As far as I'm concerned "User Agent: * Disallow" is saying "all robots: disallow". I don't understand where you get "nothing is disallowed" from.

  beynac 08:38 08 Apr 2006

I suggest that you contact Google and ask them what they need. I would expect it to be a text file on your website called "robots.txt" which contains:

User-agent: *

I don't want Google, or any other search engine to list my website (it is only there for me to experiment with html, css, php and other coding). Googlebots visit my site regularly but only look at my "robots.txt" file, which says "Disallow". This stops them looking any further.

  [DELETED] 08:42 08 Apr 2006

I want to allow them all.

  beynac 09:08 08 Apr 2006

I have been thinking about your post and had another look at my "robots.txt" file. I have a forward slash after the "Disallow:". I now understand where you got the "nothing is disallowed" from.

Ignore my previous posts: AndySD is right. I would still suggest that you contact Google.

This thread is now locked and can not be replied to.

Elsewhere on IDG sites

Amazon Prime Day 2018: Best Deals & Launches

Say aloha to the summertime vibes of artist Kim Sielbeck

The best Amazon Prime Day Apple deals 2018

Les meilleures séries Amazon Prime Video (2018)