Robot.txt Telling them where to go

  Proclaimer 04:35 12 Apr 2010

I have a site (under construction) that is split into two. One part is a general website and the other is a Shopping Cart.

The General Site is here click here and the Cart (a Zen Cart) is here click here in a folder called 'store'.

The Cart could act as a stand alone website but I am not so happy with that so I have built the front end onto it.

The whole site is not seamless and because I am using a cart it never will be. However, if I submit the 'Website' front end to the Search Engines will the Robots automatically find the sub folder with the Shopping Cart in it?

I was thinking I should create a Robot.txt file that tells them to go and find the Cart and index that as part of the site too but I don't know if I need it or what to put into the file for the robots to understand what I want them to do.

  Proclaimer 08:50 12 Apr 2010

although if a customer did hit the Zen-Cart first I wouldn't be too worried that they never seen the rest of the site. I am more concerned for the wee robots right now. :P

I do plan to link back to the Website from the Cart but I haven't really started editing the Cart yet, but thanks for pointing out that it will require links both ways.

I thought I might change the name of the Cart 'Home' button to something else and put my own Home button on the Cart that goes to the Website Homepage. Just need to work out what is a logical name for both 'Home' Buttons now.

  Ansolan 22:55 17 Apr 2010


Great advice from forum member, websites are for people not robots. Incidentally, you can't use a robots.txt file to tell robots what to index, only where you don't want them to visit.

Even that is no guarantee of not having a URL indexed but you are in any event looking for the opposite. Would be worth submitting a .xml sitemap which includes the cart URLs click here No guarantee of indexing but can help with discovery.

Beyond that the point made about internal linking is important, as is the site itself. A minor observation on that, the stuffed keywords tag. Doesn't matter in an algorithmic sense, completely ignored due to previous abuse but if the site ever happened to come up for human review, the sort of silly thing that can tip decisions if someone's having a bad day.

A far more vital point, the content. Seems to apply in other places but as an example:

click here

click here

click here

Not much good I'm afraid, pot luck which might return in searches but in general, the longer standing and/or stronger site. You need to rewrite the descriptions or preferably add much more original useful content to each one. Otherwise a common scenario is that your pages will be indexed for afew weeks/months until Google get a grip on what they represent, then many will no longer appear in the index.

  Ansolan 23:03 17 Apr 2010

Looks like the copyscape links are non functional, you can run the pages yourself anyway. The product content is all over the web.

This thread is now locked and can not be replied to.

Elsewhere on IDG sites

Samsung Galaxy S9 review: Hands-on

The art of 'British' pulp fiction

Best password managers for Mac

TV & streaming : comment regarder le Tournoi des Six Nations 2018 ?