 |
||||
| Latest | Forums | Rules | Resources | 
|
|
||||||||||
|
|||||||
| GoogleCommunity Sponsor |
Use coupon "forum" for 50% Off! |
 |
|
|
LinkBack | Thread Tools | Display Modes |
10-08-2007, 06:36 PM
|
#1 (permalink) |
|
Noogle
 Join Date: Oct 2007
Posts: 7
Thanks: 0
Thanked 0 Times in 0 Posts
 |
Robot.txt
I guess I don't understand the robot.txt file.
Is it neccessary to have and how do I set it up? Is there generic code I can use that will work and allow all of my site to be crawled? |
|
|
|
Sponsored Links
|
|
|
10-09-2007, 02:54 AM
|
#2 (permalink) |
|
Elite Googler
 
Join Date: Oct 2006
Location: Ireland
Posts: 1,085
Thanks: 0
Thanked 2 Times in 2 Posts
|
robots.txt - To allow crawling for your web site? You don't need it for this purpose. Google will crawl your web site without it.
You will need it if you want to restrict page crawling. |
|
|
|
|
| Thread Tools | |
| Display Modes | |
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| Googlebot requests robot.txt over 100 times a day | zthomasz | Search Engine Optimization Forum | 0 | 07-02-2007 03:12 PM |
| Need to know google robot. | ammarin | All About Google | 4 | 05-31-2007 09:42 AM |
| Google search engine changed the robot s | montyauto | Search Engine Optimization Forum | 6 | 05-14-2007 11:41 AM |
| A question about robot.txt | anversli | Search Engine Optimization Forum | 5 | 01-23-2006 07:11 PM |
| A question about robot.txt | anversli | Search Engine Optimization Forum | 5 | 01-01-1970 01:14 PM |
Linear Mode
