In out, In out, shake it all about...

[kwikbreaks]

I'm mystified by the search engine behaviour for one of my sites.

It's never been a top site but has been well indexed and produced fair traffic for a number of years. Early in March I noticed traffic levels had fallen to about a quarter of normal volumes. A quick investigation showed it wasn't showing up in google on any of the normal search phrases it does well with. A quick search around here made me think I'd maybe breached the cloaking rules somehow. It's a paged site and I have javascript to frame any orphan pages.

I rewrote the site to remove paging and adjusted the sitemap to suit. I also ditched some rather old and largely irrelevant pages that had been hanging around from years back when I first produced the site. I've thoroughly checked the site and humans will see exactly what the bot does - no hidden text keywords or anything of that nature (I've never used those anyway). A while later traffic increased to normal levels but only for a day. After another slack week it was back again - this time for three days and now again it isn't showing up on normally good search phrases.

Throughout this time the bot has still been crawling, the google toolbar still shows the same pagerank (4), a site: search shows all the expected pages, and a serach on the sitename still shows it top (as I would expect).

Any suggestions what is going on please?

[alex-party]

Very interesting article....
Same happened to me once and I also wonder why?!?
A newly born web site was on top for 9 months, then it was under construction, but contents remained untouched, but it just disappeared from search results and I could not find it anymore between 1 and 1000 results!
So, if someone knows why this is happening to web sites, we'd like to hear from you! Thanks.

[tms]

Its a great post. Now it makes a lot of sense.
Thanks OP

[kwikbreaks]

I have seen newly indexed sites disappear for short periods of 3 or 4 days and reappear at least twice but never anything like this. It may have happened I suppose as the site in question is very seasonal and I tend not to monitor it much during the winter months.

I worked up my own theory for newly indexed sites which may or may not be correct...

Google may run a "newly indexed" database which gets populated and used in parallel with the main index (indices?).

Periodically that could be taken offline and replaced with an empty but filling "newly indexed" database.

The original "newly indexed" database gets merged with a copy of the main index to form a new main index. While that is going on the site is invisible.

When the merge is complete the newly built main index replaces the original, the site reappears, and the cycle repeats.

[alex-party]

Interesting theory!
By the way, I don't know if there is any connection, but usually it helps to resubmit sitemap in GoogleSitemaps, then Google indexes all sites in one go, re-indexing their database for particular domain and updating Internal & External Links. That's working for me every time I do that, so if you think your web site poorly indexed, try this method, it may help.
As to in-outs, still not sure.....

[kwikbreaks]

Thanks for the idea but I did do that when I removed the old pages and framing.

It seems to be indexed well enough but just doesn't rank well on search phrases that used to put it in the top ten. But this keeps changing - a week or so out then a few days in, then back out again. As I said it makes no sense to me. I hope it settles down to back in soon as my busy period is just starting - I have paid punters and adsense money at stake

[teeny]

This has also happened to me in the past and I couldn't understand why - but then after a few months my site magically appeared again - very strange...

[zeruel]

Very interesting discussion... Maybe the problem is with google with regards to newly indexed sites... I don't have any idea on this.. I just hope this won't happen to me... and if so, surely a big thing to investigate.