| |
anversli
Junior Googler
Joined: 28 Oct 2005
1489.95 GC$
Items
|
 Posted: Sun Nov 06, 2005 9:05 am Post subject: A question about robot.txt |
 |
|
|
Hi, do i have to make a robot.txt file to get indexed on a search engine, or can i run my website without a robot.txt file.
If it's necessary how do i have to make a robot.txt file, maybe a simple example pls.
_________________
Free Online Games
Ringtones |
|
| Back to top |
|
|
Lupine1647
The One with Wolves
Joined: 14 Nov 2004
Location: Kansas City, Missouri
22324.60 GC$
Items
|
|
| Back to top |
|
|
anversli
Junior Googler
Joined: 28 Oct 2005
1489.95 GC$
Items
|
|
| Back to top |
|
|
bentong
Google Freak
Joined: 22 Mar 2005
4834.00 GC$
Items
|
|
| Back to top |
|
|
Kila_Bite
Noogle
Joined: 11 Nov 2005
Location: UK
178.00 GC$
Items
|
| Posted: Sat Nov 12, 2005 6:40 am Post subject: |
|
|
|
Robots.txt is a way of telling search engines not to go to certian parts of your web site. A suitable robots.txt file for you would be
| Code: | User-Agent: *
Allow: /
Disallow: /cgi-bin |
It usually made for people whom are likely to have big sites and want to conserve bandwidth but its also good for filtering out duplicate content too. For instance I run a Vbulletin commnuity, the Vbulletin software naturally has a lot of duplicate content, so my robots file looks like:
| Code: | User-agent: *
Disallow: /admincp/
Disallow: /ajax.php
Disallow: /attachments/
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /images/
Disallow: /includes/
Disallow: /install/
Disallow: /modcp/
Disallow: /subscriptions/
Disallow: /customavatars/
Disallow: /customprofilepics/
Disallow: /announcement.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /external.php
Disallow: /faq.php
Disallow: /frm_attach
Disallow: /image.php
Disallow: /index.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php?
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /payment_gateway.php
Disallow: /payments.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /showpost.php
Disallow: /subscription.php
Disallow: /usercp.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php |
Pretty big I know heh.
_________________
Final Fantasy, Anime and Gaming community including fully functioning Vbulletin Forums with RPG Inferno, Custom User Pages, and the Inferno Quiz System.
http://www.ff-net.org |
|
| Back to top |
|
|
physik3r
Googler
Joined: 23 Jan 2006
3358.70 GC$
Items
|
|
| Back to top |
|
|
Sponsored Links
|
| Posted: 4 Dec 2008 12:27 am Post subject: Advertisements |
|
|
|
|
|
|
| Back to top |
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
Host your free forums with Invision Plus.net forum web hosting with your own subdomain.
alexisBlue v1.2 // Theme Created By: Andrew Charron // Icons in Part By: Travis Carden |
|
|
|
|
