robots google

  FOEYDO 18:12 07 Apr 2006
Locked

Hi, I have uploaded an xml sitemap to Google and they say my robots file is 404 error, I do not have a robots file as I want no exclusions, how do I write a file that syas this? Also how long do you think it takes to get indexed in Google?Thanks

  AndySD 21:06 07 Apr 2006

Use Notepad

type in

User-agent: *

Disallow:

save the file as

robots.txt

then upload this file to the website.

  beynac 22:45 07 Apr 2006

Isn't this the wrong way round? Your text file stops robots, I thought that FOEYDO was asking for a way to encourage them.

  AndySD 23:35 07 Apr 2006

User-agent: *

The asterisk (*) in the User-agent field is shorthand for "all robots".

Disallow:


Because nothing is disallowed, everything is allowed.


It allows all robots to crawl all the files and directories.

  beynac 07:17 08 Apr 2006

I think that we will have to agree to differ. As far as I'm concerned "User Agent: * Disallow" is saying "all robots: disallow". I don't understand where you get "nothing is disallowed" from.

  beynac 08:38 08 Apr 2006

I suggest that you contact Google and ask them what they need. I would expect it to be a text file on your website called "robots.txt" which contains:

User-agent: *
Allow:

I don't want Google, or any other search engine to list my website (it is only there for me to experiment with html, css, php and other coding). Googlebots visit my site regularly but only look at my "robots.txt" file, which says "Disallow". This stops them looking any further.

  FOEYDO 08:42 08 Apr 2006

I want to allow them all.

  beynac 09:08 08 Apr 2006

I have been thinking about your post and had another look at my "robots.txt" file. I have a forward slash after the "Disallow:". I now understand where you got the "nothing is disallowed" from.

FOEYDO:
Ignore my previous posts: AndySD is right. I would still suggest that you contact Google.

This thread is now locked and can not be replied to.

Elsewhere on IDG sites

The Evil Within 2 review-in-progress

Photoshop CC 2018 released with new Curvature Pen and better brush tools

Camera tips to take better iPhone photos

Les meilleures applications de covoiturage 2017