A question about robot.txt

[anversli]

Hi, do i have to make a robot.txt file to get indexed on a search engine, or can i run my website without a robot.txt file.

If it's necessary how do i have to make a robot.txt file, maybe a simple example pls.

[Lupine1647]

You don't have to make one, but you should have one anyways. If you don't need to disallow anything just leave it blank.

[anversli]

ok thanx Lupine1647

[bentong]

Originally Posted by anversli

If it's necessary how do i have to make a robot.txt file, maybe a simple example pls.

http://www.searchengineworld.com/rob...s_tutorial.htm

good luck!

[Kila_Bite]

Robots.txt is a way of telling search engines not to go to certian parts of your web site. A suitable robots.txt file for you would be

 User-Agent: *
Allow: /
Disallow: /cgi-bin

It usually made for people whom are likely to have big sites and want to conserve bandwidth but its also good for filtering out duplicate content too. For instance I run a Vbulletin commnuity, the Vbulletin software naturally has a lot of duplicate content, so my robots file looks like:

User-agent: *
Disallow: /admincp/
Disallow: /ajax.php
Disallow: /attachments/
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /images/
Disallow: /includes/
Disallow: /install/
Disallow: /modcp/
Disallow: /subscriptions/
Disallow: /customavatars/
Disallow: /customprofilepics/
Disallow: /announcement.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /external.php
Disallow: /faq.php
Disallow: /frm_attach
Disallow: /image.php
Disallow: /index.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php?
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /payment_gateway.php
Disallow: /payments.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /showpost.php
Disallow: /subscription.php
Disallow: /usercp.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php

Pretty big I know heh.

[physik3r]

IMO this is a must-have if you're hardcore about SEO. With two lines of code I can cut the number of outbound links that are spidered on my sites by about 60%