How to avoid search engines crawling

Hit man · 03-23-2007, 02:32 AM

Hi friends,

I have 2 sites , 1 is developing site and another is developed site. I need to transfer my information from developing site to developed site so within this period i don't like any search engines(google,yahoo.msn,aol........) crawl my developing site.

Kindly tell me the way to avoid or stop search engines crawling on my site

Answer would be appreciated greatly.......

alex-party · 03-23-2007, 02:46 AM

Add this line to your page header:
<meta name="robots" content="noindex,nofollow">

It tells search robots NOT to index your page and NOT to follow links on your page.
You can also create robots.txt file with links that SHOULD NOT BE crawled and upload it to your web root directory.

Jaguar15 · 03-23-2007, 04:07 AM

If you don't want search engines to index any file of your Web site, use the following on your robots.txt:
User-agent: *
Disallow: /

Candy · 04-04-2007, 10:33 PM

To avoid undesirable search listings, webmasters can instruct spiders not to ... addressing technical issues that may keep search engines from crawling

manager2006 · 04-09-2007, 11:36 AM

Quote:

Originally Posted by Jaguar15

If you don't want search engines to index any file of your Web site, use the following on your robots.txt:
User-agent: *
Disallow: /

Yes , it is the better way to use robots.txt.

teeny · 04-09-2007, 03:24 PM

Try this link for more info:

http://www.searchtools.com/robots/robots-txt.html

Also, you might not want to block all search engines and you can just block certain search engines.

Good luck

Teeny

sharon · 04-10-2007, 04:34 AM

It is simple.
To prevent all robots from indexing a page on your site, you'd place the following meta tag into the <HEAD> section of your page:
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

tms · 04-10-2007, 11:47 AM

sharon's way is the simpliest way to do it. I tried and worked for me

03-23-2007, 02:32 AM	#1 (permalink)
Hit man Senior Googler Join Date: Mar 2007 Location: Bangalore Posts: 124 Thanks: 0 Thanked 0 Times in 0 Posts	How to avoid search engines crawling Hi friends, I have 2 sites , 1 is developing site and another is developed site. I need to transfer my information from developing site to developed site so within this period i don't like any search engines(google,yahoo.msn,aol........) crawl my developing site. Kindly tell me the way to avoid or stop search engines crawling on my site Answer would be appreciated greatly....... Last edited by Hit man; 03-23-2007 at 02:34 AM. Reason: modification

03-23-2007, 02:46 AM	#2 (permalink)
alex-party Elite Googler Join Date: Oct 2006 Location: Ireland Posts: 1,085 Thanks: 0 Thanked 2 Times in 2 Posts	Add this line to your page header: <meta name="robots" content="noindex,nofollow"> It tells search robots NOT to index your page and NOT to follow links on your page. You can also create robots.txt file with links that SHOULD NOT BE crawled and upload it to your web root directory. Last edited by alex-party; 03-23-2007 at 03:10 AM.

04-10-2007, 11:47 AM	#8 (permalink)
tms Senior Googler Join Date: Mar 2007 Posts: 126 Thanks: 0 Thanked 1 Time in 1 Post	robot.txt sharon's way is the simpliest way to do it. I tried and worked for me

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Is Search Engine Crawling Considered as hit ?	mayavi	Web Browsers and Statistics	2	10-07-2006 03:47 AM
How do I avoid snap back to search box?	mevans42	All About Google	2	07-22-2006 01:58 PM
What do Search Engines look for?	shahabr	Search Engine Optimization Forum	1	04-03-2006 11:36 PM
how much triffic are you getting from search engines?	gurufocus	All About Google	0	06-10-2005 10:00 AM
Top 10 search engines for the U.K.	geekerati	General Discussion	7	03-18-2005 09:54 AM

03-23-2007, 04:07 AM	#3 (permalink)
Jaguar15 Googler Join Date: Feb 2007 Posts: 80 Thanks: 0 Thanked 1 Time in 1 Post	If you don't want search engines to index any file of your Web site, use the following on your robots.txt: User-agent: * Disallow: /

04-04-2007, 10:33 PM	#4 (permalink)
Candy Googler Join Date: Mar 2007 Posts: 60 Thanks: 0 Thanked 0 Times in 0 Posts	To avoid undesirable search listings, webmasters can instruct spiders not to ... addressing technical issues that may keep search engines from crawling

04-09-2007, 03:24 PM	#6 (permalink)
teeny Googler Join Date: Jan 2006 Posts: 82 Thanks: 0 Thanked 0 Times in 0 Posts	Try this link for more info: http://www.searchtools.com/robots/robots-txt.html Also, you might not want to block all search engines and you can just block certain search engines. Good luck Teeny

04-10-2007, 04:34 AM	#7 (permalink)
sharon Noogle Join Date: Apr 2007 Posts: 11 Thanks: 0 Thanked 0 Times in 0 Posts	It is simple. To prevent all robots from indexing a page on your site, you'd place the following meta tag into the <HEAD> section of your page: <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

GoogleCommunity Sponsor
Use coupon "forum" for 50% Off!


Sponsored Links