Robots.txt help

dop · 02-04-2007, 08:04 PM

Is this a good robots.txt?

#User-agent: lycra
#Disallow: /

#User-agent: *
#Disallow: /tmp
#Disallow: /logs

User-agent: *
Disallow:

What does the
User-agent: *
Disallow:
tell the spiders? That all are allowed or none?

If you haven't figured it out by now, I am working with another company to help them. Just going over their stuff. I always thought the above should read

User-agent: *
Disallow: /

To allow all others....

nenesio · 03-04-2007, 09:30 PM

no, don`t use dillow/

intelliot · 03-04-2007, 11:00 PM

The main reason to use robots.txt is to prevent your site from getting indexed by search engines. Without a robots.txt file (or with a blank one), they will assume you allow it.

nenesio · 03-05-2007, 12:02 AM

sorry, my commentary was imcomplet
i say to put of final all the folders

disallow/

that prohibits to the search to enter in of
index (front page)

leftybogs · 03-05-2007, 06:37 AM

Quote:

Originally Posted by nenesio

sorry, my commentary was imcomplet
i say to put of final all the folders

disallow/

that prohibits to the search to enter in of
index (front page)

you can specify what page in your site that you don't want to indexed by SEs spiders...

nenesio · 03-05-2007, 11:39 AM

ok thist is my robots.txt

User-agent: Googlebot*
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /editor/
Disallow: /includes/
Disallow: /language/
Disallow: /mambots/
Disallow: /media/
Disallow: /modules/
Disallow: /templates/
Disallow: /installation/

User-agent: hot*
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /editor/
Disallow: /includes/
Disallow: /language/
Disallow: /mambots/
Disallow: /media/
Disallow: /modules/
Disallow: /templates/
Disallow: /installation/

but it use of this bloquiaria form acesso to googlebot in all the page

User-agent: Googlebot*
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /editor/
Disallow: /includes/
Disallow: /language/
Disallow: /mambots/
Disallow: /media/
Disallow: /modules/
Disallow: /templates/
Disallow: /installation/

User-agent: hot*
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /editor/
Disallow: /includes/
Disallow: /language/
Disallow: /mambots/
Disallow: /media/
Disallow: /modules/
Disallow: /templates/
Disallow: /installation/

User-agent:*
Disallow:/ <----- block the index

In order to eliminate all the pages that are within a directory:
User-agent: Googlebot
Disallow: /Directory

In order to eliminate all the archives of a type of specific file, for example .gif, will have to use the following entrance of robots.txt:
User-agent: Googlebot
Disallow: /*.gif$

In order to eliminate generated pages of dynamic way, it will have to use this entrance of robots.txt:
User-agent: Googlebot
Disallow: /*?

you need but information?
http://www.google.com/support/webmas...y?answer=35303

WebGeek182 · 03-06-2007, 02:49 PM

Quote:

Originally Posted by nenesio

sorry, my commentary was imcomplet
i say to put of final all the folders

disallow/

that prohibits to the search to enter in of
index (front page)

No, that disallows your whole site from being indexed. Be careful with that.

nenesio · 03-06-2007, 09:12 PM

reads my other post, and to understand which I say

Jaguar15 · 03-07-2007, 04:04 AM

Also try to look at this for some tips on robots
www[.]pageresource[.]com/zine/robotstxt[.]htm

nenesio · 03-07-2007, 07:50 AM

that page gives the reason me

02-04-2007, 08:04 PM	#1 (permalink)
dop Noogle Join Date: Feb 2007 Posts: 10 Thanks: 0 Thanked 0 Times in 0 Posts	Robots.txt help Is this a good robots.txt? #User-agent: lycra #Disallow: / #User-agent: * #Disallow: /tmp #Disallow: /logs User-agent: * Disallow: What does the User-agent: * Disallow: tell the spiders? That all are allowed or none? If you haven't figured it out by now, I am working with another company to help them. Just going over their stuff. I always thought the above should read User-agent: * Disallow: / To allow all others....

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
robots.txt question	z4dll	All About Google	3	06-12-2006 07:08 AM
Creating robots	cruocitae	Chit Chat	6	05-15-2006 11:17 PM
Google Robots	geekerati	Chit Chat	1	01-16-2006 08:21 AM
Robots.txt	matias	All About Google	5	08-30-2005 11:23 PM
how google robots?	malapati	All About Google	1	11-09-2004 11:00 PM

03-04-2007, 09:30 PM	#2 (permalink)
nenesio Noogle Join Date: Jan 2007 Posts: 5 Thanks: 0 Thanked 0 Times in 0 Posts	no, don`t use dillow/

03-04-2007, 11:00 PM	#3 (permalink)
intelliot Google Guru Join Date: May 2004 Posts: 2,528 Thanks: 15 Thanked 128 Times in 34 Posts	The main reason to use robots.txt is to prevent your site from getting indexed by search engines. Without a robots.txt file (or with a blank one), they will assume you allow it.

03-05-2007, 12:02 AM	#4 (permalink)
nenesio Noogle Join Date: Jan 2007 Posts: 5 Thanks: 0 Thanked 0 Times in 0 Posts	sorry, my commentary was imcomplet i say to put of final all the folders disallow/ that prohibits to the search to enter in of index (front page)

03-05-2007, 11:39 AM	#6 (permalink)
nenesio Noogle Join Date: Jan 2007 Posts: 5 Thanks: 0 Thanked 0 Times in 0 Posts	ok thist is my robots.txt User-agent: Googlebot* Disallow: /administrator/ Disallow: /cache/ Disallow: /components/ Disallow: /editor/ Disallow: /includes/ Disallow: /language/ Disallow: /mambots/ Disallow: /media/ Disallow: /modules/ Disallow: /templates/ Disallow: /installation/ User-agent: hot* Disallow: /administrator/ Disallow: /cache/ Disallow: /components/ Disallow: /editor/ Disallow: /includes/ Disallow: /language/ Disallow: /mambots/ Disallow: /media/ Disallow: /modules/ Disallow: /templates/ Disallow: /installation/ but it use of this bloquiaria form acesso to googlebot in all the page User-agent: Googlebot* Disallow: /administrator/ Disallow: /cache/ Disallow: /components/ Disallow: /editor/ Disallow: /includes/ Disallow: /language/ Disallow: /mambots/ Disallow: /media/ Disallow: /modules/ Disallow: /templates/ Disallow: /installation/ User-agent: hot* Disallow: /administrator/ Disallow: /cache/ Disallow: /components/ Disallow: /editor/ Disallow: /includes/ Disallow: /language/ Disallow: /mambots/ Disallow: /media/ Disallow: /modules/ Disallow: /templates/ Disallow: /installation/ User-agent:* Disallow:/ <----- block the index In order to eliminate all the pages that are within a directory: User-agent: Googlebot Disallow: /Directory In order to eliminate all the archives of a type of specific file, for example .gif, will have to use the following entrance of robots.txt: User-agent: Googlebot Disallow: /.gif$ In order to eliminate generated pages of dynamic way, it will have to use this entrance of robots.txt: User-agent: Googlebot Disallow: /? you need but information? http://www.google.com/support/webmas...y?answer=35303

03-06-2007, 09:12 PM	#8 (permalink)
nenesio Noogle Join Date: Jan 2007 Posts: 5 Thanks: 0 Thanked 0 Times in 0 Posts	reads my other post, and to understand which I say

03-07-2007, 04:04 AM	#9 (permalink)
Jaguar15 Googler Join Date: Feb 2007 Posts: 80 Thanks: 0 Thanked 1 Time in 1 Post	Also try to look at this for some tips on robots www[.]pageresource[.]com/zine/robotstxt[.]htm

03-07-2007, 07:50 AM	#10 (permalink)
nenesio Noogle Join Date: Jan 2007 Posts: 5 Thanks: 0 Thanked 0 Times in 0 Posts	that page gives the reason me

GoogleCommunity Sponsor


Sponsored Links